AWS Batch: Overnight Long-Running Coding Agent Jobs Without Lambda Timeouts

Your agent can plan a monorepo migration in thirty seconds and then spend four hours applying it: clone, branch, edit hundreds of files, run tests, open a PR. That work does not belong in Lambda (15 minutes), Express workflows (5 minutes), or a lonely EC2 you forgot to terminate. AWS Batch is the production shape for overnight coding-agent jobs — job definitions with container images, queues with priority, spot for cheap bulk work, on-demand for deadline runs, and retries that do not invent homemade schedulers. Use EventBridge Scheduler to fire the batch; use Batch to run it.

⚡ TL;DR: Put multi-hour agent work (migrations, large refactors, eval sweeps) on AWS Batch. Keep interactive tool calls on Lambda + Step Functions. Prefer spot for rewindable jobs, on-demand for customer-facing SLAs. Schedule with EventBridge Scheduler; sandbox builds still fit CodeBuild when you need a short CI-shaped box.

Why Lambda and Express are the wrong ceiling

Lambda Recursive Loop Protection exists because teams keep self-invoking past the wall — see Lambda Recursive Loop Protection. Batch removes the need to fake longevity with recursion.

// ❌ "Chunk forever" anti-pattern — Lambda chain pretending to be a job
export async function migrateChunk(event: { cursor: string; depth: number }) {
  if (event.depth > 200) throw new Error("too deep");
  const next = await migrateSomeFiles(event.cursor);
  // Invoke self again... until concurrency and cost cry
}

# ✅ Submit one Batch job; agent loop lives inside the container
aws batch submit-job \
  --job-name "agent-migrate-acme-$(date +%Y%m%d)" \
  --job-queue agent-overnight-queue \
  --job-definition agent-long-runner:7 \
  --parameters tenant=acme,repo=github.com/acme/monorepo,mode=migrate

Job definition for a long-running coding agent

Package the agent runtime as a container: Bedrock/Converse client, git, language toolchains, and a clear entrypoint that exits non-zero on fatal tool failures.

{
  "jobDefinitionName": "agent-long-runner",
  "type": "container",
  "platformCapabilities": ["EC2"],
  "containerProperties": {
    "image": "123456789012.dkr.ecr.us-east-1.amazonaws.com/agent-runner:1.4.2",
    "vcpus": 4,
    "memory": 8192,
    "jobRoleArn": "arn:aws:iam::123456789012:role/agent-batch-job",
    "executionRoleArn": "arn:aws:iam::123456789012:role/ecsTaskExecutionRole",
    "environment": [
      { "name": "AGENT_MODE", "value": "overnight" },
      { "name": "BEDROCK_REGION", "value": "us-east-1" }
    ],
    "secrets": [
      {
        "name": "GITHUB_TOKEN",
        "valueFrom": "arn:aws:secretsmanager:us-east-1:123456789012:secret:agents/github"
      }
    ],
    "logConfiguration": {
      "logDriver": "awslogs",
      "options": {
        "awslogs-group": "/aws/batch/agent-long-runner",
        "awslogs-region": "us-east-1",
        "awslogs-stream-prefix": "job"
      }
    }
  },
  "retryStrategy": {
    "attempts": 2,
    "evaluateOnExit": [
      { "onStatusReason": "Host EC2*", "action": "RETRY" },
      { "onExitCode": "0", "action": "EXIT" },
      { "onExitCode": "*", "action": "EXIT" }
    ]
  },
  "timeout": { "attemptDurationSeconds": 28800 }
}

✅ Retry on spot reclaim / host issues; ❌ do not blindly retry application exit 1 from a bad prompt — that burns money.

IAM on agent-batch-job should follow the same least-privilege story as interactive tools: no env-file long-lived keys; prefer KMS decrypt grants and Secrets Manager refs.

Queues, compute environments, spot vs on-demand

Separate interactive capacity from overnight capacity.

# Spot CE for rewindable agent jobs
aws batch create-compute-environment \
  --compute-environment-name agent-spot-ce \
  --type MANAGED \
  --compute-resources type=EC2,allocationStrategy=SPOT_CAPACITY_OPTIMIZED,\
minvCpus=0,maxvCpus=256,subnets=subnet-aaa,securityGroupIds=sg-bbb,\
instanceRole=ecsInstanceRole,bidPercentage=100

Design the agent to checkpoint (PR branch pushed, DynamoDB cursor, S3 artifact) so a spot reclaim is a resume, not a restart from zero. For CI-shaped one-shot sandboxes that are not multi-hour agent loops, CodeBuild spec sandboxes remain simpler.

Scheduling overnight without cron drift

Do not put cron(0 2 * * ? *) on a random EC2. Use EventBridge Scheduler to batch:SubmitJob with a role that can only submit to the overnight queue.

# Scheduler target payload sketch (Batch SubmitJob)
{
  "JobName": "agent-nightly-<tenant>",
  "JobQueue": "agent-overnight-queue",
  "JobDefinition": "agent-long-runner",
  "Parameters": {
    "tenant": "<tenant>",
    "mode": "migrate"
  }
}

Timezone and one-time vs recurring schedules are covered in EventBridge Scheduler overnight batches — Batch is the executor, Scheduler is the clock.

Orchestration: Batch inside vs beside Step Functions

• Beside: Scheduler → Batch for the heavy job; Step Functions for interactive graphs. Clearest ops story.
• Inside: Standard workflow task that submits Batch and waits (arn:aws:states:::batch:submitJob.sync) when human approval gates the overnight run.
• Not: Express wrapping a four-hour Batch wait.

Keep Step Functions agent graphs for tool fan-out; keep Batch for wall-clock work that would otherwise violate Lambda timeouts.

Checklist: first overnight agent on Batch

• [ ] Container image with agent runtime + toolchains; non-zero exit on fatal errors
• [ ] Job definition with 4–8h attempt timeout, spot-aware retries
• [ ] Separate overnight queue from interactive Lambda concurrency
• [ ] Checkpoint to git/S3/DynamoDB before long tool bursts
• [ ] EventBridge Scheduler submit (no crontab on pets)
• [ ] CloudWatch Logs group + Insights fields (tenant_id, job_id) for forensics
• [ ] Kill switch / feature flag still readable from Batch (AppConfig or SSM)

Long-running agents are a job system problem. Batch already is one — stop teaching Lambda to pretend.