7 views
Coding agents fail in ways dashboards cannot ssh into: a half-applied patch, a stuck npm install, a mystery process holding the workspace lock. The wrong fix is a standing SSH key on every Fargate/EC2 sandbox. AWS Systems Manager Session Manager gives operators an IAM-gated shell through the SSM agent — no inbound 22, optional session encryption, and logs to S3/CloudWatch that actually survive the “who typed rm -rf?” review. Distinct from Network Firewall egress filtering (what the sandbox can call) and IAM Roles Anywhere (how on-prem runners get AWS creds): Session Manager is human break-glass with an audit trail.
⚡ TL;DR: Install/enable SSM agent on sandbox hosts (or ECS exec where appropriate), deny SSH security-group ingress, grant ssm:StartSession to an on-call role with MFA, enable session logging to a locked S3 bucket, and alert on every break-glass via CloudTrail. Related: CloudTrail Lake IAM abuses, Fargate Spot sandboxes, CodeBuild sandboxes, GuardDuty compromised creds.
Why SSH keys rot on agent fleets
| Approach | Blast radius | Audit quality |
|---|---|---|
| Shared SSH key in Secrets Manager | Key leak = every sandbox | Weak (maybe bastion logs) |
| Per-host SSH + bastion | Bastion becomes VIP target | Medium |
| SSM Session Manager | IAM + MFA per session | Strong (CloudTrail + session logs) |
| ECS Exec / kubectl exec | Task/pod scoped | Good if logging enabled |
Prerequisites on the sandbox
bash
Prefer VPC interface endpoints for # ✅ EC2 / Bottlerocket / Amazon Linux — SSM agent + instance profile
# Instance profile needs AmazonSSMManagedInstanceCore (or tighter custom)
aws ec2 associate-iam-instance-profile \
--instance-id i-0abc \
--iam-instance-profile Name=coding-agent-sandbox-ssm
# Security group: egress for SSM endpoints (or VPC interface endpoints);
# ✅ no inbound 22 from 0.0.0.0/0
aws ec2 revoke-security-group-ingress \
--group-id sg-sandbox \
--protocol tcp --port 22 --cidr 0.0.0.0/0 || true
ssm, ssmmessages, ec2messages so sandboxes with locked egress (Network Firewall) can still accept Sessions without public SSM API access.
For ECS/Fargate, use ECS Exec (Session Manager under the hood) with enableExecuteCommand and task IAM — document which path your fleet uses so on-call runbooks do not invent SSH.
Start an audited session
bash
# ✅ MFA-protected on-call role assumed first
aws sts assume-role \
--role-arn arn:aws:iam::123456789012:role/coding-agent-breakglass \
--role-session-name oncall-$(date +%s) \
--serial-number arn:aws:iam::123456789012:mfa/you \
--token-code 123456
aws ssm start-session --target i-0abc
# ECS Exec equivalent
aws ecs execute-command \
--cluster agent-sandboxes \
--task TASK_ARN \
--container sandbox \
--interactive \
--command "/bin/bash"
json
❌ {
"Version": "2012-10-17",
"Statement": [
{
"Sid": "BreakGlassSession",
"Effect": "Allow",
"Action": ["ssm:StartSession"],
"Resource": [
"arn:aws:ec2:*:*:instance/*",
"arn:aws:ssm:*:*:document/AWS-StartSSHSession",
"arn:aws:ssm:*:*:document/SSM-SessionManagerRunShell"
],
"Condition": {
"StringEquals": {
"ssm:resourceTag/workload": "coding-agent",
"aws:MultiFactorAuthPresent": "true"
},
"Bool": { "aws:MultiFactorAuthPresent": "true" }
}
},
{
"Sid": "TerminateOwn",
"Effect": "Allow",
"Action": ["ssm:TerminateSession", "ssm:ResumeSession"],
"Resource": "arn:aws:ssm:*:*:session/${aws:userid}-*"
}
]
}
ssm:StartSession on * without tag conditions or MFA — that is a lateral-movement gift.
Session logging that holds up in IR
bash
Example prefs fragment:
# ✅ Session Manager preferences → S3 bucket + CloudWatch Logs
# Bucket: Block Public Access, Object Lock optional for IR retention,
# deny delete for breakglass role
aws ssm update-document \
--name SSM-SessionManagerRunShell \
--document-version '$LATEST' \
--content file://session-prefs.json
json
Send CloudTrail {
"schemaVersion": "1.0",
"description": "coding-agent break-glass logging",
"sessionType": "Standard_Stream",
"inputs": {
"s3BucketName": "coding-agent-session-logs",
"s3KeyPrefix": "ssm/",
"s3EncryptionEnabled": true,
"cloudWatchLogGroupName": "/ssm/coding-agent-sessions",
"cloudWatchEncryptionEnabled": true,
"idleSessionTimeout": "15",
"maxSessionDuration": "60",
"runAsEnabled": true,
"runAsDefaultUser": "ssm-user"
}
}
StartSession events into CloudTrail Lake and page on-call when sessions open outside change windows. Pair with Detective if the same principal also spikes STS anomalies.
Runbook: break-glass without making it normal
- Confirm agent run ID / task ARN and customer impact
- Assume break-glass role with MFA; note ticket ID in session reason (tags / chat)
start-session/ ECS Exec — read-only first (ls, logs); write only with peer ack- Capture evidence to IR bucket; do not “fix prod” inside the sandbox if promote-path exists
- Terminate session; rotate any secrets touched; open postmortem stub
bash
# ✅ detect sessions via CloudTrail Lake (sketch)
# SELECT userIdentity.arn, eventTime, requestParameters
# FROM cloudtrail_lake
# WHERE eventName = 'StartSession'
# AND eventTime > date_add('hour', -24, current_timestamp)
Failure modes
| Issue | Symptom | Mitigation |
|---|---|---|
| Agent missing | Target not in Fleet Manager | Bake agent into AMI; AL2023 usually has it; check instance profile |
| Endpoints blocked | Session hangs | Interface endpoints + firewall allowlist |
| Logs disabled | “We SSHed but no tape” | SCPs / Config rule requiring logging prefs |
| Standing access | Break-glass becomes daily driver | Time-boxed IAM + Access Analyzer review (Access Analyzer) |
| Shared ssm-user abuse | Attribution mush | runAs to named users; prefer per-operator identity |
Production checklist
- [ ] No inbound SSH to sandbox security groups
- [ ] SSM agent + instance/task role with least privilege
- [ ] VPC endpoints for SSM in locked-egress accounts
- [ ] Break-glass role: MFA, tags, short session max, idle timeout
- [ ] Session logs to encrypted S3 + CloudWatch; retain per IR policy
- [ ] CloudTrail alert on
StartSession/ExecuteCommand - [ ] Runbook distinguishes read-only vs mutate; ticket ID required
- [ ] Quarterly access review; remove standing humans from sandbox roles
- [ ] Document ECS Exec vs EC2 Session Manager per environment
FAQ
Q: Does Session Manager work on Fargate?A: Use ECS Exec for Fargate tasks (Session Manager plumbing). Classic
start-session to instance IDs is the EC2 path. Do not assume one CLI covers both.
Q: Is this a replacement for Network Firewall?A: No. Firewall constrains egress. Session Manager constrains and audits human ingress. You want both. Q: What about CodeBuild sandboxes?
A: Prefer ephemeral CodeBuild with logs in CloudWatch; break-glass is rarer. If you need a shell into a long-lived builder host, treat it like EC2 SSM — same IAM + logging bar.
Related reading
- AWS Network Firewall: Egress Filtering for Coding-Agent Sandboxes
- AWS IAM Roles Anywhere: Cert-Based Creds for On-Prem Runners
- CloudTrail Lake: Query Agent IAM Abuses
- AWS Fargate Spot: Cheap Ephemeral Sandboxes
Watch: 100 Days of AI on YouTube
Short videos from CheatCoders, one AI engineering topic per day.
Most viewed
- Python Decorators Explained: From Simple Wrappers to Production Patterns
- AI Agent Frameworks in 2025: LangGraph vs CrewAI vs AutoGen vs Raw API
- Claude Code vs Cursor vs GitHub Copilot: Choose by Workflow, and Set Up One Repo That Works With All Three
- Python String Methods: Every str Method With Real Production Examples
- API Debugging in 30 Seconds: The Fastest Ways to Fix API Issues
Newly added
- n8n vs AI Agent: I Gave Both the Same Job (Local, Ollama)
- Local AI Video Generator: Build Your Own App (Free, Python, No GPU)
- AI Wrote Raw Binary Instead of Code. Here’s What Happened (WebAssembly)
- I Built a Browser That Only Runs Encrypted Binary (Ep 1)
- RAG From Scratch: Chunking, Embeddings, Reranking and Evaluation (Real Numbers)
Deep-dive PDF
Get the expanded guide for this post — extra diagrams-style checklists, failure modes, and a production walkthrough. Free when you subscribe to CheatCoders.
Already subscribed? or open the subscribe page.
Discover more from CheatCoders
Subscribe to get the latest posts sent to your email.