AWS Fargate Spot: Cheap Ephemeral Sandboxes for Coding Agents

Your coding agent just spun a full Fargate task on on-demand capacity to clone a repo, run pytest, and apply a 40-line patch — then sat idle for nine minutes waiting for the next tool turn. Multiply that by 80 tenants and your sandbox bill looks like a product P&L. Fargate Spot is the unfair advantage for short-lived clone/test/apply sandboxes: you pay Spot prices for capacity that should die in minutes, not hours. Pair with CodeBuild spec sandboxes when you want managed build images, and with AWS Budgets + Cost Anomaly so Spot savings do not hide a runaway fleet — this post is the ephemeral Spot sandbox layer.

⚡ TL;DR: Prefer Fargate Spot for agent sandboxes that clone → test → apply → exit in under ~15 minutes. Handle SIGTERM (2-minute notice) by checkpointing patch state to S3/DynamoDB, never leaving half-applied trees as source of truth. Keep task roles least-privilege; complement with Organizations SCPs. Related: CodeBuild sandboxes, ECR+Lambda containers, multi-tenant Fargate Spot runtimes.

Why always-on Fargate loses to Spot for agent sandboxes

Coding-agent sandboxes are bursty and discardable:

1. Clone (or mount) a worktree
2. Run lint/tests/typecheck
3. Apply a patch or open a PR artifact
4. Exit — the next turn may land on a different task

Always-on Fargate (or oversized CodeBuild fleets) optimizes for warm disks and warm caches. Agents optimize for isolation per turn and cheap failure. Spot capacity matches that shape if you treat interruption as a first-class signal, not an incident.

Task definitions that stay Spot-friendly

Keep the task thin. Bake toolchains into the image (ECR container pattern applies mentally even when the runtime is ECS). Pass work via env + S3, not giant command lines.

// ✅ Spot capacity provider strategy for agent sandboxes (CDK)
import * as ecs from "aws-cdk-lib/aws-ecs";

const cluster = new ecs.Cluster(this, "AgentSandboxes", { vpc });
cluster.enableFargateCapacityProviders();

const taskDef = new ecs.FargateTaskDefinition(this, "SandboxTask", {
  cpu: 1024,
  memoryLimitMiB: 2048,
  runtimePlatform: {
    cpuArchitecture: ecs.CpuArchitecture.ARM64,
    operatingSystemFamily: ecs.OperatingSystemFamily.LINUX,
  },
});

taskDef.addContainer("sandbox", {
  image: ecs.ContainerImage.fromEcrRepository(repo, "agent-sandbox-1.4.2"),
  logging: ecs.LogDrivers.awsLogs({ streamPrefix: "agent-sandbox" }),
  environment: {
    WORK_BUCKET: workBucket.bucketName,
    // ❌ Never bake GitHub PATs here — use Secrets Manager (next post)
  },
  secrets: {
    GITHUB_TOKEN: ecs.Secret.fromSecretsManager(ghSecret, "token"),
  },
  stopTimeout: cdk.Duration.seconds(120), // align with Spot SIGTERM window
});

// RunTask with Spot
await ecs.send(
  new RunTaskCommand({
    cluster: cluster.clusterName,
    taskDefinition: taskDef.taskDefinitionArn,
    capacityProviderStrategy: [
      { capacityProvider: "FARGATE_SPOT", weight: 1, base: 0 },
      // ✅ optional on-demand fallback for critical tenants
      { capacityProvider: "FARGATE", weight: 0, base: 0 },
    ],
    networkConfiguration: {
      awsvpcConfiguration: {
        subnets: privateSubnets,
        securityGroups: [sandboxSg],
        assignPublicIp: "DISABLED",
      },
    },
    overrides: {
      containerOverrides: [
        {
          name: "sandbox",
          environment: [
            { name: "TENANT_ID", value: tenantId },
            { name: "TURN_ID", value: turnId },
            { name: "PATCH_S3_URI", value: patchUri },
          ],
        },
      ],
    },
  })
);

# ❌ Spot sandbox that treats /workspace as durable source of truth
def apply_and_sleep(repo_dir: str, patch: str):
    apply_patch(repo_dir, patch)
    # agent waits for next chat turn inside the same task
    time.sleep(600)  # burning Spot + inviting interruption mid-idle

Rule: one Spot task ≈ one tool turn (or one bounded Step Functions branch). Persist artifacts to S3 with conditional writes; orchestrate multi-step graphs with Step Functions.

Interruption handling that coding agents actually need

Fargate Spot sends SIGTERM ≈ two minutes before reclaim. Your entrypoint must:

1. Stop accepting new tool work
2. Flush in-flight patch / test results to S3
3. Write a turn_status=interrupted row (DynamoDB) so the planner can retry
4. Exit 0 if checkpoint succeeded (so orchestrators do not double-apply blindly)

#!/usr/bin/env bash
# ✅ entrypoint with Spot-aware drain
set -euo pipefail
CHECKPOINT=""
on_term() {
  echo "spot_sigterm turn=$TURN_ID"
  if [[ -n "${CHECKPOINT}" ]]; then
    aws s3 cp "$CHECKPOINT" "s3://$WORK_BUCKET/tenants/$TENANT_ID/turns/$TURN_ID/checkpoint.tgz"
    aws dynamodb update-item \
      --table-name AgentTurns \
      --key "{\"pk\":{\"S\":\"$TENANT_ID#$TURN_ID\"}}" \
      --update-expression "SET #s = :i, updatedAt = :t" \
      --expression-attribute-names '{"#s":"status"}' \
      --expression-attribute-values "{\":i\":{\"S\":\"interrupted\"},\":t\":{\"S\":\"$(date -u +%FT%TZ)\"}}"
  fi
  exit 0
}
trap on_term TERM

git clone --depth 1 "$REPO_URL" /work/repo
cd /work/repo
aws s3 cp "$PATCH_S3_URI" /tmp/patch.diff
git apply /tmp/patch.diff
pytest -q | tee /tmp/test.out
tar -czf /tmp/out.tgz /tmp/test.out
CHECKPOINT=/tmp/out.tgz
aws s3 cp /tmp/out.tgz "s3://$WORK_BUCKET/tenants/$TENANT_ID/turns/$TURN_ID/result.tgz"

❌ Ignoring SIGTERM and relying on ECS to hard-kill leaves half-written trees and duplicate PR comments when Step Functions retries.

IAM and blast radius on Spot fleets

Spot makes it cheap to run more tasks. That amplifies bad IAM:

• Task role: s3:GetObject/PutObject on tenants/${tenant}/* only
• Deny iam:*, organizations:*, broad ec2:* via SCPs
• Authorize which tools may RunTask with Verified Permissions / Cedar
• Kill switches via AppConfig when Spot interruption rates spike in a region

// ✅ task-role fragment — tenant-prefixed objects only
{
  "Effect": "Allow",
  "Action": ["s3:GetObject", "s3:PutObject"],
  "Resource": "arn:aws:s3:::agent-work-prod/tenants/${aws:PrincipalTag/tenant_id}/*"
}

Spot vs CodeBuild vs Batch — pick by duration

• < 60s tools → Lambda (+ SnapStart / containers), not Fargate
• 1–15 min clone/test/apply → Fargate Spot (this post)
• Spec-driven CI image, one-shot → CodeBuild sandboxes
• Hours-long eval / overnight → AWS Batch (Spot compute env OK)

Do not force Spot on tasks that must finish a 40-minute integration suite with zero retry budget — use on-demand Fargate or Batch with retries.

Production checklist

• [ ] Capacity provider prefers FARGATE_SPOT; on-demand only as explicit fallback
• [ ] Task stopTimeout / entrypoint trap handles SIGTERM and checkpoints to S3
• [ ] Turn state machine records interrupted and retries idempotently
• [ ] Secrets from Secrets Manager / SSM — never env baked into image
• [ ] Task role tenant-scoped; SCP + Cedar deny privilege escalation
• [ ] Budgets/anomaly alarms on ECS Fargate Spot usage per tenant OU
• [ ] Packages via CodeArtifact, not public npm during Spot storms
• [ ] Logs/traces tagged tenant_id, turn_id, capacity_provider=FARGATE_SPOT

FAQ

Q: Will Spot interruptions ruin agent UX?
A: Only if a single task owns a multi-minute chat session. Scope tasks to one turn, checkpoint, and let the planner retry — users feel a retry, not a stuck IDE.

Q: Can I mix Spot and on-demand in one service?
A: Yes — weight Spot high, keep a base of on-demand for premium tenants. Encode the choice in the orchestrator, not in tribal knowledge.

Q: Is this the same as the older “isolated Fargate Spot runtimes” post?
A: That piece covers multi-tenant isolation patterns; this one is specifically cheap ephemeral sandboxes + interruption + task defs for clone/test/apply loops.

Fargate Spot turns coding-agent sandboxes from a standing army into a militia: cheap, short-lived, and interruption-aware. Wire SIGTERM to checkpoints, keep IAM boring, and let Step Functions / Batch own anything longer than a turn.