Ephemeral Debug Sidecars: Audited Access Without Permanent Prod Shells

Ephemeral Debug Sidecars: Audited Access Without Permanent Prod Shells
12 views
Permanent SSH on production ECS/EKS tasks is a gift to attackers and a compliance headache. Break-glass still matters during sev-1s. Ephemeral debug sidecars give you a time-boxed, audited shell adjacent to the workload—without baking OpenSSH into every image forever.
⚡ TL;DR: Ship tasks without SSH. On break-glass, mutate the task definition or pod to add a debug sidecar (or use ECS Exec / kubectl debug) with IAM-gated SSM, command logging, and a hard TTL. Revoke automatically. Pair with Secure AI Sandboxes, Human-in-the-Loop Gates, and Node Core Dump Analysis.

Ephemeral Debug Sidecars: production guidance

text
Prod task definition:
 - app container only
 - readonlyRootFilesystem where possible
 - no SSH daemon, no shared host keys
Break-glass:
 - dual-control ticket + change ID
 - enable ECS Exec OR attach debug sidecar for N minutes
 - session logs to CloudWatch / S3
 - auto-disable after TTL
✅ ECS Exec / ephemeral sidecar with audit.
❌ Long-lived bastion keys copied into task roles “just in case.”

ECS Exec with session logging

json
{
 "family": "checkout-api",
 "enableExecuteCommand": true,
 "containerDefinitions": [{
 "name": "app",
 "image": "123.dkr.ecr.us-east-1.amazonaws.com/checkout:sha-abc",
 "linuxParameters": { "initProcessEnabled": true }
 }]
}
bash
# Break-glass (after approval)
aws ecs execute-command \
 --cluster prod \
 --task "$TASK_ARN" \
 --container app \
 --interactive \
 --command "/bin/sh"

# Require SSM Session Manager preferences: S3/CloudWatch logging + KMS
Gate who can call ecs:ExecuteCommand with IAM conditions on cluster/service tags and require MFA. Dual-control for production matches Human-in-the-Loop Gates.

Ephemeral debug sidecar pattern (EKS)

yaml
# kubectl debug / ephemeral container sketch
apiVersion: v1
kind: Pod
metadata:
 name: checkout-7f9
spec:
 ephemeralContainers:
 - name: debugger
 image: 123.dkr.ecr.us-east-1.amazonaws.com/debug-toolbox:2026.09
 targetContainerName: app
 securityContext:
 allowPrivilegeEscalation: false
 readOnlyRootFilesystem: true
 runAsNonRoot: true
 env:
 - name: DEBUG_TTL_SECONDS
 value: "1800"
Toolbox images should be minimal, signed, and free of standing credentials. Prefer kubectl debug over mutating the primary container. For Node segfaults, copy cores out via audited channels—see Node Core Dump Analysis.

Automation: TTL and revoke

typescript
// Step Functions / EventBridge: disable execute-command after TTL
export async function revokeExec(cluster: string, service: string) {
 // update service to enableExecuteCommand=false
 // notify ticket; require new approval to re-enable
}
Record: who requested, approver, task ARN, session IDs, commands (from SSM logs), and revoke time. Treat AI agents the same—no unsupervised prod shells; sandboxes stay ephemeral as in Secure AI Sandboxes.

Closing checklist

  • [ ] Prod images ship without SSH daemons
  • [ ] ECS Exec / ephemeral debug requires MFA + dual-control
  • [ ] Session Manager logging to immutable storage with KMS
  • [ ] Hard TTL auto-revokes execute permissions
  • [ ] Debug toolbox images are signed and minimal
  • [ ] Incidents link session IDs to change tickets

Related reading

Last updated on September 11, 2026 Related: Python’s slots Secret: Cut Memory 60% and Speed Up Attribute Access by 35%; Debugging Production Postgres Slow Queries Without Causing Downtime; Kubernetes for Developers: The Concepts You Must Know to Deploy and Debug in K8s.

Watch: 100 Days of AI on YouTube

Short videos from CheatCoders, one AI engineering topic per day.

Open the playlist on YouTube · CheatCoders on YouTube

Deep-dive PDF

Get the expanded guide for this post — extra diagrams-style checklists, failure modes, and a production walkthrough. Free when you subscribe to CheatCoders.

Already subscribed? or open the subscribe page.


Discover more from CheatCoders

Subscribe to get the latest posts sent to your email.

Comments

No comments yet. Why don’t you start the discussion?

Leave a comment

No account needed. Name and email are optional.