AWS Batch for Coding Agents: Parallel Overnight PR Reviews and Long-Running Jobs Without Lambda Timeouts

21 views
You have a backlog of open pull requests and a repository scan that should be finished before standup. Running the agent from a laptop means someone babysits it. A single Lambda function stops at 15 minutes, which is not enough for a deep review that clones a large repository and runs tests. Starting EC2 instances by hand works until someone forgets to stop one. AWS Batch is built for this kind of work. You describe the job once (a container image, CPU, memory, roles), put work on a queue, and Batch provisions compute, runs the jobs, retries the ones that fail for infrastructure reasons, and scales back down when the queue is empty. This guide covers two shapes of coding-agent work on Batch: wide fleets (one child job per pull request or repository, using array jobs) and long single jobs (a migration that runs for hours). It replaces our earlier post on long-running Batch jobs, so both patterns now live here.
TL;DR
  • Use a Fargate or Fargate Spot compute environment unless you need GPUs, a custom AMI, more than 32 vCPUs or more than 244 GiB of memory per job.
  • Submit one array job (size 2 to 10,000) instead of thousands of single jobs. Each child reads AWS_BATCH_JOB_ARRAY_INDEX to pick its pull request.
  • Always set a timeout. Batch has no default timeout, and a job that times out is not retried.
  • Write a retry strategy that exits on permanent errors. If no evaluateOnExit rule matches, Batch retries.
  • Make every child idempotent, so a retried attempt never posts the same review comment twice.

The building blocks, in plain terms

Batch has four concepts. If you know these, the rest of the service makes sense.
  • Job definition: the template for a job: container image, vCPU and memory, IAM roles, environment variables, secrets, retry strategy and timeout. Think of it as a versioned recipe.
  • Job queue: where submitted jobs wait. A queue has a priority and an ordered list of up to three compute environments.
  • Compute environment: the capacity jobs run on. Managed environments can be EC2, Spot, Fargate or Fargate Spot. A single queue can’t mix EC2-based and Fargate-based environments.
  • Job: one submitted unit of work. An array job is a parent that fans out into child jobs which share the same definition but receive different index numbers.
The AWS Batch documentation recommends Fargate for most scenarios. Fargate jobs can typically be provisioned in about 30 seconds, while a new EC2 instance usually takes a few minutes. Each Fargate job also runs in its own isolation boundary, without sharing a kernel or network interface with other jobs. That isolation matters when the job runs an AI agent that executes code from a pull request. Pick EC2 when a job needs a GPU, a custom AMI, more than 32 vCPUs or more than 244 GiB of memory, or when you run so many concurrent jobs that you hit Fargate throttling limits.
Figure 1. An overnight PR-review fleet on AWS Batch: a schedule starts a submitter, which writes a manifest and submits one array job. Diagram by CheatCoders.
Figure 1. An overnight PR-review fleet on AWS Batch: a schedule starts a submitter, which writes a manifest and submits one array job. Diagram by CheatCoders.

Wide or long: pick the shape first

ShapeExampleBatch featureWatch out for
Wide (many small units)Review every open PR, scan 400 repositoriesArray job with an S3 manifestManifest length must match the array size
Long (one big unit)Migrate a monorepo to a new frameworkSingle job with a long timeout and checkpointsSpot interruptions; Fargate jobs shouldn’t expect to run more than 14 days
Ordered pipelineScan, then open issues for findingsJob dependencies (N_TO_N, SEQUENTIAL) or Step FunctionsA job can have at most 20 dependencies
InteractiveChat or IDE requestsNot Batch; use Lambda or a serviceQueueing adds seconds to minutes of start latency

Before you start

  • A container image for your agent in Amazon ECR, with git, your language toolchains and an entrypoint that exits with meaningful codes.
  • A VPC with private subnets. Fargate jobs in subnets without a NAT gateway need assignPublicIp set to ENABLED, or VPC endpoints for the services they call (ECR, S3, CloudWatch Logs, Secrets Manager, Bedrock).
  • Two IAM roles: an execution role that lets Batch pull the image, write logs and fetch secrets, and a job role that the agent code uses (for example, bedrock:InvokeModel and access to its S3 prefix).

Step 1: Create compute environments and a queue

Create a Fargate Spot environment for cheap, interruptible capacity and an on-demand Fargate environment as a fallback. The queue tries environments in ascending order, so Spot is used first.
bash
aws batch create-compute-environment \
  --compute-environment-name agents-fargate-spot \
  --type MANAGED \
  --state ENABLED \
  --compute-resources type=FARGATE_SPOT,maxvCpus=64,subnets=subnet-aaa,subnet-bbb,securityGroupIds=sg-agents

aws batch create-compute-environment \
  --compute-environment-name agents-fargate \
  --type MANAGED \
  --state ENABLED \
  --compute-resources type=FARGATE,maxvCpus=32,subnets=subnet-aaa,subnet-bbb,securityGroupIds=sg-agents

aws batch create-job-queue \
  --job-queue-name agents-overnight \
  --state ENABLED \
  --priority 10 \
  --compute-environment-order order=1,computeEnvironment=agents-fargate-spot order=2,computeEnvironment=agents-fargate
maxvCpus is your hard concurrency ceiling. With 2 vCPUs per child, a 64 vCPU Spot environment runs at most 32 children at once. That number also bounds how fast you call Bedrock, so size it with your model quotas in mind. Fargate environments don’t accept EC2-only settings such as instanceTypes, allocationStrategy, minvCpus or bidPercentage. If you move to EC2 Spot later, use type=SPOT with the SPOT_PRICE_CAPACITY_OPTIMIZED allocation strategy, which the docs recommend over SPOT_CAPACITY_OPTIMIZED in most cases.

Step 2: Register the job definition

json
{
  "jobDefinitionName": "agent-pr-review",
  "type": "container",
  "platformCapabilities": ["FARGATE"],
  "containerProperties": {
    "image": "111122223333.dkr.ecr.ap-south-1.amazonaws.com/coding-agent:1.8.0",
    "command": ["python", "-m", "agent.review"],
    "resourceRequirements": [
      {"type": "VCPU", "value": "2"},
      {"type": "MEMORY", "value": "4096"}
    ],
    "executionRoleArn": "arn:aws:iam::111122223333:role/agents-batch-execution",
    "jobRoleArn": "arn:aws:iam::111122223333:role/agents-batch-job",
    "networkConfiguration": {"assignPublicIp": "DISABLED"},
    "fargatePlatformConfiguration": {"platformVersion": "LATEST"},
    "environment": [
      {"name": "PROMPT_VERSION", "value": "review-v3"}
    ],
    "secrets": [
      {"name": "GITHUB_TOKEN", "valueFrom": "arn:aws:secretsmanager:ap-south-1:111122223333:secret:agents/github"}
    ],
    "logConfiguration": {
      "logDriver": "awslogs",
      "options": {
        "awslogs-group": "/aws/batch/agents",
        "awslogs-region": "ap-south-1",
        "awslogs-stream-prefix": "pr-review"
      }
    }
  },
  "retryStrategy": {
    "attempts": 3,
    "evaluateOnExit": [
      {"onExitCode": "3", "action": "EXIT"},
      {"onExitCode": "4", "action": "EXIT"}
    ]
  },
  "timeout": {"attemptDurationSeconds": 1800}
}
Register it with aws batch register-job-definition --cli-input-json file://agent-pr-review.json. Three details are worth understanding:
  • Retry rules. attempts can be 1 to 10. When evaluateOnExit is set and no rule matches, Batch retries. Here the agent exits with code 3 for “bad input, don’t retry” and 4 for “policy refused”, and everything else (a Spot interruption, a crashed container, an internal error) gets another attempt. If you prefer opt-in retries, end the list with {"onReason": "*", "action": "EXIT"} as the Batch user guide shows.
  • Timeouts. Batch has no default timeout. The minimum is 60 seconds. When the time is up, the container gets SIGTERM, then SIGKILL 30 seconds later. A job terminated by its timeout is not retried, so a stuck tool loop fails fast rather than burning three attempts.
  • Secrets. The execution role needs secretsmanager:GetSecretValue on the referenced secret. The token arrives as an environment variable and never appears in the job definition.

Step 3: Submit an array job from a manifest

An array job has a size between 2 and 10,000. Batch gives each child the same job ID with its index appended (for example jobid:42) and sets AWS_BATCH_JOB_ARRAY_INDEX, starting at 0. The submitter writes the list of work to S3 first, then submits once. One SubmitJob call creates the whole fleet, which matters because SubmitJob is limited to 50 transactions per second per account.
python
import datetime
import json

import boto3

s3 = boto3.client("s3")
batch = boto3.client("batch")
BUCKET = "agents-overnight"
MAX_ARRAY = 10_000


def submit_overnight(prs: list[dict]) -> list[str]:
    """Write one manifest per chunk and submit one array job per manifest."""
    day = datetime.date.today().isoformat()
    job_ids = []
    for start in range(0, len(prs), MAX_ARRAY):
        chunk = prs[start:start + MAX_ARRAY]
        key = f"manifests/{day}/prs-{start:05d}.json"
        s3.put_object(Bucket=BUCKET, Key=key, Body=json.dumps({"prs": chunk}).encode())
        request = {
            "jobName": f"pr-review-{day}-{start:05d}",
            "jobQueue": "agents-overnight",
            "jobDefinition": "agent-pr-review",
            "containerOverrides": {
                "environment": [{"name": "MANIFEST_KEY", "value": key}]
            },
            "tags": {"workload": "coding-agent", "batch": "overnight"},
            "propagateTags": True,
        }
        if len(chunk) >= 2:
            request["arrayProperties"] = {"size": len(chunk)}
        job_ids.append(batch.submit_job(**request)["jobId"])
    return job_ids


def handler(event, context):
    prs = list_open_prs()  # your forge API call, returns [{"repo": ..., "number": ..., "sha": ...}]
    if not prs:
        return {"submitted": []}
    return {"submitted": submit_overnight(prs)}
Note the len(chunk) >= 2 check: an array of size 1 is invalid, so a single pull request becomes a normal job.

Step 4: Write an idempotent worker

Each child runs the same container. It finds its item from the index, checks whether this exact review already exists, does the work, and records completion. Because a retry can happen after a comment was posted but before the job finished, the idempotency key is written before posting, with a conditional write.
python
import json
import os
import signal
import sys

import boto3
from botocore.exceptions import ClientError

s3 = boto3.client("s3")
ledger = boto3.resource("dynamodb").Table("agent-review-ledger")
PERMANENT_FAILURE = 3  # matches the EXIT rule in the job definition

stopping = False


def on_sigterm(signum, frame):
    # Timeout or Spot interruption: finish the current step, checkpoint, exit.
    global stopping
    stopping = True


signal.signal(signal.SIGTERM, on_sigterm)


def main() -> int:
    index = int(os.environ.get("AWS_BATCH_JOB_ARRAY_INDEX", "0"))
    attempt = int(os.environ.get("AWS_BATCH_JOB_ATTEMPT", "1"))
    body = s3.get_object(Bucket="agents-overnight", Key=os.environ["MANIFEST_KEY"])["Body"].read()
    prs = json.loads(body)["prs"]
    if index >= len(prs):
        print(f"index {index} outside manifest of {len(prs)}")
        return PERMANENT_FAILURE
    pr = prs[index]
    key = f"{pr['repo']}#{pr['number']}#{pr['sha']}#{os.environ['PROMPT_VERSION']}"

    try:
        ledger.put_item(
            Item={"review_key": key, "status": "claimed", "attempt": attempt},
            ConditionExpression="attribute_not_exists(review_key) OR #s <> :done",
            ExpressionAttributeNames={"#s": "status"},
            ExpressionAttributeValues={":done": "posted"},
        )
    except ClientError as err:
        if err.response["Error"]["Code"] == "ConditionalCheckFailedException":
            print(f"{key} already posted; skipping")
            return 0
        raise

    review = run_agent_review(pr, should_stop=lambda: stopping)  # your agent loop
    if review is None:
        return 1  # interrupted: non-zero, not 3, so Batch retries
    post_review_comment(pr, review, external_id=key)  # forge call keyed by external_id
    ledger.update_item(
        Key={"review_key": key},
        UpdateExpression="SET #s = :done",
        ExpressionAttributeNames={"#s": "status"},
        ExpressionAttributeValues={":done": "posted"},
    )
    return 0


if __name__ == "__main__":
    sys.exit(main())

Step 5: Schedule it in your time zone

EventBridge Scheduler is the clock and Batch is the executor. Schedules accept a time zone, so “2 AM in Bengaluru” stays 2 AM without UTC arithmetic. The schedule invokes the submitter Lambda from Step 3.
bash
aws scheduler create-schedule \
  --name agents-overnight-pr-review \
  --schedule-expression "cron(0 2 * * ? *)" \
  --schedule-expression-timezone "Asia/Kolkata" \
  --flexible-time-window '{"Mode": "FLEXIBLE", "MaximumWindowInMinutes": 15}' \
  --target '{"Arn": "arn:aws:lambda:ap-south-1:111122223333:function:agents-submitter", "RoleArn": "arn:aws:iam::111122223333:role/scheduler-invoke-submitter"}'

Long single jobs: migrations that run for hours

Some agent work doesn’t split. A framework migration might edit hundreds of files and then run the full test suite. Lambda’s 15-minute limit and Step Functions Express’s 5-minute limit rule those out, and chaining Lambda functions that re-invoke themselves is fragile and hard to observe. Run it as one Batch job with a timeout that fits the work (for example "attemptDurationSeconds": 28800 for eight hours) and design it to resume:
  • Push work to a branch at milestones, and store a cursor (last processed package or file) in DynamoDB or S3.
  • On start, read AWS_BATCH_JOB_ATTEMPT. If it’s greater than 1, resume from the cursor instead of starting over.
  • Handle SIGTERM by finishing the current file, saving the cursor and exiting non-zero so the retry rule applies. Remember that a timeout kill is not retried, so set the timeout generously and let your own soft deadline fire first.
When a human must approve the run, or later steps depend on the result, start the job from a Step Functions Standard workflow with the .sync integration. The state machine waits for the Batch job to finish and fails the state if the job fails:
json
{
  "Migrate": {
    "Type": "Task",
    "Resource": "arn:aws:states:::batch:submitJob.sync",
    "Arguments": {
      "JobName": "agent-migrate-monorepo",
      "JobQueue": "arn:aws:batch:ap-south-1:111122223333:job-queue/agents-overnight",
      "JobDefinition": "arn:aws:batch:ap-south-1:111122223333:job-definition/agent-long-runner:4",
      "Timeout": {"AttemptDurationSeconds": 28800}
    },
    "Next": "OpenPullRequest"
  }
}

Worked example: 180 pull requests overnight

Here’s how one night plays out with the setup above (the numbers are illustrative):
  1. At 02:00 IST the schedule invokes the submitter. It finds 180 open pull requests, writes manifests/2026-10-06/prs-00000.json and submits one array job of size 180.
  2. Each child asks for 2 vCPUs, so the 64 vCPU Spot environment runs up to 32 children at a time. The queue spills to on-demand Fargate only when Spot capacity can’t be placed.
  3. Child 17 loses its Spot capacity mid-review. No retry rule matches, so Batch moves it back to RUNNABLE for attempt 2. The ledger shows the review was claimed but never posted, so the new attempt runs the review again and posts exactly once.
  4. Child 63 belongs to a pull request whose branch was deleted. The worker exits with code 3, which hits the EXIT rule, so there is no wasted retry.
  5. Child 99 gets stuck in a tool loop. At 30 minutes Batch sends SIGTERM, the worker stops, and the attempt fails without a retry.
  6. Because two children failed, the parent array job ends as FAILED once every child reaches a terminal state. An EventBridge rule on Batch job state change events sends the failed child IDs to your team channel, and the morning shift reruns just those two pull requests.

Troubleshooting

SymptomLikely causeFix
Jobs sit in RUNNABLENo capacity can be placed: maxvCpus reached, the job asks for more than the environment allows, or the environment is invalidRead statusReason in DescribeJobs; consider jobStateTimeLimitActions on the queue to cancel stuck jobs
Container never starts (image pull fails)Private subnet without NAT or VPC endpoints, and assignPublicIp is DISABLEDAdd a NAT gateway or ECR, S3 and Logs endpoints
SubmitJob rejects the array sizeSize is 1 or above 10,000Submit single items as normal jobs; chunk large manifests
Same bad PR retried three timesNo EXIT rule for permanent errorsExit with a dedicated code and match it with onExitCode
A stuck job was never retriedIt was terminated by its timeout, which Batch doesn’t retryAdd an in-app soft deadline that exits non-zero before the hard timeout
Throttling errors from the submitterMany single SubmitJob calls (50 TPS limit)Use one array job per manifest
Secret not found at startExecution role lacks secretsmanager:GetSecretValueGrant it on the secret ARN to the execution role, not the job role
Can’t add a compute environment to the queueMixing EC2 and Fargate environments, or more than threeUse separate queues per capacity type

Production checklist

  • ☐ Fargate or Fargate Spot compute environments with a deliberate maxvCpus
  • ☐ Job definition pins an image tag or digest; secrets come from Secrets Manager through the execution role
  • ☐ Timeout set on every job; in-app soft deadline fires first
  • ☐ Retry strategy with explicit EXIT rules for permanent errors
  • ☐ Array jobs fed by an S3 manifest; single items submitted as normal jobs
  • ☐ Idempotency ledger so retries never duplicate comments or pull requests
  • ☐ Job role scoped to the agent’s S3 prefix and the Bedrock models it needs
  • ☐ Tags (workload, batch) propagated for cost reports, plus an AWS Budgets alert
  • ☐ EventBridge rule on job state changes for failure alerts

FAQ

Fargate or EC2 compute for coding agents?

Fargate for most agents: less to patch, per-job isolation and faster starts. Choose EC2 for GPUs, custom AMIs, jobs above 32 vCPUs or 244 GiB, or very high concurrency.

Why not Step Functions Map with Lambda?

It works well when every item finishes within Lambda’s 15 minutes and needs no heavy toolchain. Large checkouts, test suites and multi-hour work fit Batch better. You can combine them: Step Functions for the workflow, Batch for the heavy steps.

How is Batch different from CodeBuild for this?

CodeBuild is shaped around builds and buildspecs. Batch is a general job scheduler with queues, priorities, array jobs and retry rules, which suits fan-out fleets better.

What happens if I cancel the parent array job?

All child jobs are cancelled or terminated with it. You can also cancel individual children without affecting the rest, but the parent will then end as FAILED.

Does Spot make sense for agent work?

Yes, for work that can be retried safely. Idempotent children and checkpointed long jobs turn an interruption into a retry, not lost work. Keep deadline-critical jobs on on-demand capacity.

Official documentation

Related on CheatCoders: SSM Parameter Store for agent config · Athena for agent audit logs Last updated on October 6, 2026

Watch: 100 Days of AI on YouTube

Short videos from CheatCoders, one AI engineering topic per day.

Open the playlist on YouTube · CheatCoders on YouTube

Deep-dive PDF

Get the expanded guide for this post — extra diagrams-style checklists, failure modes, and a production walkthrough. Free when you subscribe to CheatCoders.

Already subscribed? or open the subscribe page.


Discover more from CheatCoders

Subscribe to get the latest posts sent to your email.

Comments

No comments yet. Why don’t you start the discussion?

Leave a comment

No account needed. Name and email are optional.