12 views
Permanent SSH on production ECS/EKS tasks is a gift to attackers and a compliance headache. Break-glass still matters during sev-1s. Ephemeral debug sidecars give you a time-boxed, audited shell adjacent to the workload—without baking OpenSSH into every image forever.
⚡ TL;DR: Ship tasks without SSH. On break-glass, mutate the task definition or pod to add a debug sidecar (or use ECS Exec / kubectl debug) with IAM-gated SSM, command logging, and a hard TTL. Revoke automatically. Pair with Secure AI Sandboxes, Human-in-the-Loop Gates, and Node Core Dump Analysis.
Ephemeral Debug Sidecars: production guidance
text
✅ ECS Exec / ephemeral sidecar with audit.Prod task definition:
- app container only
- readonlyRootFilesystem where possible
- no SSH daemon, no shared host keys
Break-glass:
- dual-control ticket + change ID
- enable ECS Exec OR attach debug sidecar for N minutes
- session logs to CloudWatch / S3
- auto-disable after TTL
❌ Long-lived bastion keys copied into task roles “just in case.”
ECS Exec with session logging
json
{
"family": "checkout-api",
"enableExecuteCommand": true,
"containerDefinitions": [{
"name": "app",
"image": "123.dkr.ecr.us-east-1.amazonaws.com/checkout:sha-abc",
"linuxParameters": { "initProcessEnabled": true }
}]
}
bash
Gate who can call # Break-glass (after approval)
aws ecs execute-command \
--cluster prod \
--task "$TASK_ARN" \
--container app \
--interactive \
--command "/bin/sh"
# Require SSM Session Manager preferences: S3/CloudWatch logging + KMS
ecs:ExecuteCommand with IAM conditions on cluster/service tags and require MFA. Dual-control for production matches Human-in-the-Loop Gates.
Ephemeral debug sidecar pattern (EKS)
yaml
Toolbox images should be minimal, signed, and free of standing credentials. Prefer # kubectl debug / ephemeral container sketch
apiVersion: v1
kind: Pod
metadata:
name: checkout-7f9
spec:
ephemeralContainers:
- name: debugger
image: 123.dkr.ecr.us-east-1.amazonaws.com/debug-toolbox:2026.09
targetContainerName: app
securityContext:
allowPrivilegeEscalation: false
readOnlyRootFilesystem: true
runAsNonRoot: true
env:
- name: DEBUG_TTL_SECONDS
value: "1800"
kubectl debug over mutating the primary container. For Node segfaults, copy cores out via audited channels—see Node Core Dump Analysis.
Automation: TTL and revoke
typescript
Record: who requested, approver, task ARN, session IDs, commands (from SSM logs), and revoke time. Treat AI agents the same—no unsupervised prod shells; sandboxes stay ephemeral as in Secure AI Sandboxes.
// Step Functions / EventBridge: disable execute-command after TTL
export async function revokeExec(cluster: string, service: string) {
// update service to enableExecuteCommand=false
// notify ticket; require new approval to re-enable
}
Closing checklist
- [ ] Prod images ship without SSH daemons
- [ ] ECS Exec / ephemeral debug requires MFA + dual-control
- [ ] Session Manager logging to immutable storage with KMS
- [ ] Hard TTL auto-revokes execute permissions
- [ ] Debug toolbox images are signed and minimal
- [ ] Incidents link session IDs to change tickets
Related reading
- Secure AI Sandboxes: Ephemeral ECS Tasks for Agent Tool Execution
- Human-in-the-Loop Gates: Dual Control for Prod-Touching Agent Tools
- Node Core Dump Analysis: Symbolicate Native Crashes Inside Containers
- Policy-as-Code for AI: OPA Checks Before Every Shell Execution
Watch: 100 Days of AI on YouTube
Short videos from CheatCoders, one AI engineering topic per day.
Most viewed
- Python Decorators Explained: From Simple Wrappers to Production Patterns
- Claude Code vs Cursor vs GitHub Copilot: Choose by Workflow, and Set Up One Repo That Works With All Three
- AI Agent Frameworks in 2025: LangGraph vs CrewAI vs AutoGen vs Raw API
- Python String Methods: Every str Method With Real Production Examples
- Css Animation Performance: Performance-Optimized Motion for Modern UIs
Newly added
- AWS Systems Manager Parameter Store: Hierarchical Runtime Config for Coding Agents
- Amazon Cognito Identity Pools: Federate Coding-Agent Tool Callers Without Long-Lived Keys
- Amazon Athena for Coding Agents: Query Tool Audit Logs and Eval Traces on S3 Without a Warehouse
- AWS Batch for Coding Agents: Parallel Overnight PR Reviews and Long-Running Jobs Without Lambda Timeouts
- Amazon Bedrock Prompt Management: Version and A/B Test Coding-Agent System Prompts
Deep-dive PDF
Get the expanded guide for this post — extra diagrams-style checklists, failure modes, and a production walkthrough. Free when you subscribe to CheatCoders.
Already subscribed? or open the subscribe page.
Discover more from CheatCoders
Subscribe to get the latest posts sent to your email.
