Ephemeral Debug Sidecars: Audited Access Without Permanent Prod Shells

Permanent SSH on production ECS/EKS tasks is a gift to attackers and a compliance headache. Break-glass still matters during sev-1s. Ephemeral debug sidecars give you a time-boxed, audited shell adjacent to the workload—without baking OpenSSH into every image forever.

⚡ TL;DR: Ship tasks without SSH. On break-glass, mutate the task definition or pod to add a debug sidecar (or use ECS Exec / kubectl debug) with IAM-gated SSM, command logging, and a hard TTL. Revoke automatically. Pair with Secure AI Sandboxes, Human-in-the-Loop Gates, and Node Core Dump Analysis.

Default posture: no standing shells

Prod task definition:
  - app container only
  - readonlyRootFilesystem where possible
  - no SSH daemon, no shared host keys
Break-glass:
  - dual-control ticket + change ID
  - enable ECS Exec OR attach debug sidecar for N minutes
  - session logs to CloudWatch / S3
  - auto-disable after TTL

✅ ECS Exec / ephemeral sidecar with audit.
❌ Long-lived bastion keys copied into task roles “just in case.”

ECS Exec with session logging

{
  "family": "checkout-api",
  "enableExecuteCommand": true,
  "containerDefinitions": [{
    "name": "app",
    "image": "123.dkr.ecr.us-east-1.amazonaws.com/checkout:sha-abc",
    "linuxParameters": { "initProcessEnabled": true }
  }]
}

# Break-glass (after approval)
aws ecs execute-command \
  --cluster prod \
  --task "$TASK_ARN" \
  --container app \
  --interactive \
  --command "/bin/sh"

# Require SSM Session Manager preferences: S3/CloudWatch logging + KMS

Gate who can call ecs:ExecuteCommand with IAM conditions on cluster/service tags and require MFA. Dual-control for production matches Human-in-the-Loop Gates.

Ephemeral debug sidecar pattern (EKS)

# kubectl debug / ephemeral container sketch
apiVersion: v1
kind: Pod
metadata:
  name: checkout-7f9
spec:
  ephemeralContainers:
  - name: debugger
    image: 123.dkr.ecr.us-east-1.amazonaws.com/debug-toolbox:2026.09
    targetContainerName: app
    securityContext:
      allowPrivilegeEscalation: false
      readOnlyRootFilesystem: true
      runAsNonRoot: true
    env:
    - name: DEBUG_TTL_SECONDS
      value: "1800"

Toolbox images should be minimal, signed, and free of standing credentials. Prefer kubectl debug over mutating the primary container. For Node segfaults, copy cores out via audited channels—see Node Core Dump Analysis.

Automation: TTL and revoke

// Step Functions / EventBridge: disable execute-command after TTL
export async function revokeExec(cluster: string, service: string) {
  // update service to enableExecuteCommand=false
  // notify ticket; require new approval to re-enable
}

Record: who requested, approver, task ARN, session IDs, commands (from SSM logs), and revoke time. Treat AI agents the same—no unsupervised prod shells; sandboxes stay ephemeral as in Secure AI Sandboxes.

Closing checklist

• [ ] Prod images ship without SSH daemons
• [ ] ECS Exec / ephemeral debug requires MFA + dual-control
• [ ] Session Manager logging to immutable storage with KMS
• [ ] Hard TTL auto-revokes execute permissions
• [ ] Debug toolbox images are signed and minimal
• [ ] Incidents link session IDs to change tickets

Related reading

• Secure AI Sandboxes: Ephemeral ECS Tasks for Agent Tool Execution
• Human-in-the-Loop Gates: Dual Control for Prod-Touching Agent Tools
• Node Core Dump Analysis: Symbolicate Native Crashes Inside Containers
• Policy-as-Code for AI: OPA Checks Before Every Shell Execution

Last updated on September 11, 2026