Bedrock Batch Inference for Overnight Refactors: Shard, Run, Validate and Apply in Daytime PRs

Some code changes are huge but boring: migrating every call of a deprecated SDK client, rewriting import paths after a package rename, swapping one logging API for another. Doing them through an interactive coding agent means thousands of request-response round trips during working hours, a single giant diff at the end, and a reviewer who gives up halfway through.

Amazon Bedrock batch inference is built for the first half of that problem. You put many prompts in JSONL files in Amazon S3, submit one job, and Bedrock writes the responses back to S3 when it’s done. Nobody waits on each answer. This guide shows a workflow that uses it for repo-wide refactors: shard the repository, run the batch overnight, validate every proposed change offline, and merge the survivors during the day as small stacked pull requests. It merges our earlier “batch inference overnight” bootcamp lesson into this guide.

⚡ TL;DR

• Use batch inference for mechanical, deferrable transforms. Keep interactive Converse calls for changes that need judgment or tool use.
• Shard by package so each record is small and compile-safe. Each JSONL line has a recordId and a modelInput; with the Converse invocation type, modelInput uses the Converse request format.
• Batch jobs don’t support tool calling, and the batch inference guide says they don’t support structured output either. Ask for JSON in the prompt, then parse and validate every result yourself.
• Output order isn’t guaranteed: join on recordId. Check the job’s manifest.json.out for success and error counts.
• Never auto-merge. Apply each shard in a disposable worktree, run the type checker and tests, and open draft PRs per package.

Who this is for: platform and developer-experience engineers who own large repositories. You need access to a Bedrock model that supports batch inference in your Region, an S3 bucket, and permission to create IAM roles.

When batch beats an interactive agent

Batch inference processes each record independently, with no back-and-forth between the model and your code. That rules out anything that needs tool calls or multiple turns, and it’s exactly why it fits mechanical refactors. Use it when all of these are true:

• The change is mechanical and can be described precisely: “replace DocumentClient from aws-sdk v2 with DynamoDBDocumentClient from @aws-sdk/lib-dynamodb“.
• Each file or small group of files can be changed on its own, without reading the rest of the repo.
• A rejected shard is cheap: you can drop it and fix it interactively later.
• You can check the result automatically with a compiler, linter or tests.

Stay interactive when the change needs design decisions, behavior changes across many modules, or edits to security-sensitive configuration such as IAM policies. Those need a human in the loop at each step, not a morning review of thousands of lines.

Step 1: Shard the repository

A shard is the unit of work for one record. Keep it small enough that one bad file can’t sink a large chunk of the job, and aligned with package boundaries so each shard can be type-checked on its own.

import json
from pathlib import Path

ROOT = Path("packages")
MAX_FILES = 20          # tune so each record stays well within the model's context
NEEDLE = "aws-sdk/clients/dynamodb"


def shards():
    for pkg in sorted(p for p in ROOT.iterdir() if (p / "package.json").exists()):
        files = sorted(
            str(f) for f in pkg.rglob("*.ts")
            if "node_modules" not in f.parts and NEEDLE in f.read_text(errors="ignore")
        )
        for i in range(0, len(files), MAX_FILES):
            yield {"id": f"{pkg.name}-{i // MAX_FILES:03d}", "package": pkg.name,
                   "files": files[i:i + MAX_FILES]}


if __name__ == "__main__":
    out = list(shards())
    Path("batch").mkdir(exist_ok=True)
    Path("batch/shards.json").write_text(json.dumps(out, indent=2))
    print(f"{len(out)} shards")

Only files that actually contain the pattern become records. That keeps the job small and makes the morning diff reviewable. Record the commit SHA you sharded from; you’ll apply results against that same SHA.

Step 2: Build Converse-format JSONL

Each line of the input file is one JSON object with a recordId and a modelInput. If you create the job with modelInvocationType set to Converse, modelInput follows the Converse request body, so the same code works across models. The file must use the .jsonl extension.

import json
from pathlib import Path

SYSTEM = (
    "You migrate TypeScript files from the aws-sdk v2 DynamoDB DocumentClient to "
    "@aws-sdk/lib-dynamodb (DynamoDBDocumentClient + command classes). "
    "Change nothing else. Reply with ONLY a JSON object: "
    '{"edits": [{"path": "<file path>", "content": "<full new file content>"}], '
    '"skipped": [{"path": "<file path>", "reason": "<why>"}]}'
)


def build_input(shards_file: str, out_file: str, sha: str) -> int:
    shards = json.loads(Path(shards_file).read_text())
    with open(out_file, "w") as out:
        for s in shards:
            parts = [f"Repository commit: {sha}"]
            for f in s["files"]:
                parts.append(f"FILE {f}\n<<<\n{Path(f).read_text()}\n>>>")
            out.write(json.dumps({
                "recordId": s["id"],
                "modelInput": {
                    "system": [{"text": SYSTEM}],
                    "messages": [{"role": "user", "content": [{"text": "\n\n".join(parts)}]}],
                    "inferenceConfig": {"maxTokens": 8192, "temperature": 0},
                },
            }) + "\n")
    return len(shards)

Two design choices here. Asking for full file content instead of a diff avoids malformed hunks, at the cost of more output tokens; for small files it’s usually the more reliable option, and you produce the diff yourself in the apply step. And the system prompt explicitly allows “skipped” entries, so the model has a way to say “I’m not sure” instead of guessing.

Before you build the pipeline, check the batch inference quotas for your model in Service Quotas: there is a minimum number of records per job, a maximum number of records per file and per job, and a maximum input file size. A tiny refactor may not reach the minimum, in which case just use on-demand calls.

Step 3: Submit the job with least-privilege roles

Two identities are involved. The caller (your CI job) needs bedrock:CreateModelInvocationJob (plus get, list and stop for monitoring), scoped to the model and job ARNs, and permission to pass the service role. The service role is what Bedrock assumes to read your input and write your output. Its trust policy allows bedrock.amazonaws.com, ideally with aws:SourceAccount and aws:SourceArn conditions, and its permissions policy allows s3:GetObject, s3:PutObject and s3:ListBucket on the input and output locations. If you run the job through an inference profile, the service role also needs bedrock:InvokeModel on the profile and the models it routes to.

import os
import time

import boto3

s3 = boto3.client("s3")
bedrock = boto3.client("bedrock")

BUCKET = os.environ["BATCH_BUCKET"]


def submit(run_id: str, local_jsonl: str) -> str:
    key = f"refactors/{run_id}/input/records.jsonl"
    s3.upload_file(local_jsonl, BUCKET, key)
    job = bedrock.create_model_invocation_job(
        jobName=f"refactor-{run_id}",
        roleArn=os.environ["BATCH_SERVICE_ROLE_ARN"],
        modelId=os.environ["REFACTOR_MODEL_ID"],
        modelInvocationType="Converse",
        inputDataConfig={"s3InputDataConfig": {"s3Uri": f"s3://{BUCKET}/refactors/{run_id}/input/"}},
        outputDataConfig={"s3OutputDataConfig": {"s3Uri": f"s3://{BUCKET}/refactors/{run_id}/output/"}},
        timeoutDurationInHours=24,
        tags=[{"key": "workload", "value": "repo-refactor"}],
    )
    return job["jobArn"]


def wait(job_arn: str, poll_seconds: int = 300) -> str:
    """For CI convenience; in production prefer EventBridge job state notifications."""
    while True:
        status = bedrock.get_model_invocation_job(jobIdentifier=job_arn)["status"]
        if status in ("Completed", "PartiallyCompleted", "Failed", "Stopped", "Expired"):
            return status
        time.sleep(poll_seconds)

Tag the job so batch spend shows up separately from interactive usage in your cost reports. Put the model ID, prompt version and commit SHA in your own run metadata as well, so you can reproduce a run. Note that batch inference isn’t supported for provisioned models, so point the job at an on-demand model or a supported inference profile.

Step 4: Parse and validate the results

When the job finishes, the output prefix contains one output file per input JSONL file and a manifest.json.out with record and token counts. Every output line carries the recordId and either a modelOutput or an error. Output order isn’t guaranteed, so key everything by recordId.

import json

import boto3

s3 = boto3.client("s3")


def load_results(bucket: str, prefix: str) -> dict:
    """Return {recordId: {"edits": [...], "skipped": [...]} | {"error": "..."}}."""
    results = {}
    pages = s3.get_paginator("list_objects_v2").paginate(Bucket=bucket, Prefix=prefix)
    for page in pages:
        for obj in page.get("Contents", []):
            key = obj["Key"]
            if not key.endswith(".out") or key.endswith("manifest.json.out"):
                continue
            body = s3.get_object(Bucket=bucket, Key=key)["Body"].read().decode()
            for line in body.splitlines():
                rec = json.loads(line)
                rid = rec["recordId"]
                if "error" in rec:
                    results[rid] = {"error": json.dumps(rec["error"])[:300]}
                    continue
                results[rid] = parse_model_json(rec["modelOutput"])
    return results


def parse_model_json(model_output: dict) -> dict:
    """Converse-format output: output.message.content[].text holds the JSON reply."""
    blocks = model_output["output"]["message"]["content"]
    text = "".join(b.get("text", "") for b in blocks).strip()
    start, end = text.find("{"), text.rfind("}")
    try:
        data = json.loads(text[start:end + 1])
    except ValueError:
        return {"error": "reply was not valid JSON"}
    if not isinstance(data.get("edits"), list):
        return {"error": "missing edits list"}
    for e in data["edits"]:
        if not isinstance(e.get("path"), str) or not isinstance(e.get("content"), str):
            return {"error": "edit without path/content"}
    return data

Reject anything that isn’t valid JSON or doesn’t match the expected shape. Also reject edits whose path wasn’t part of that shard; a model that “helpfully” edits another file has left the mechanical lane.

Step 5: Apply in daytime, behind CI gates

This is the part the old “overnight AI refactor” posts tended to skip, and it’s the part that keeps main green. Batch output is a set of proposals. Each shard is applied in its own worktree from the original commit, compiled and tested, and only then turned into a draft PR.

#!/usr/bin/env bash
# apply_shard.sh <shard-id> <package> <base-sha>
set -euo pipefail
SHARD="$1"; PKG="$2"; BASE="$3"
WT="/tmp/wt-${SHARD}"

git worktree add -B "ai/refactor-${SHARD}" "$WT" "$BASE"
python3 tools/write_edits.py --shard "$SHARD" --root "$WT"   # writes the validated edits

cd "$WT"
git diff --stat
pnpm install --frozen-lockfile
pnpm --filter "./packages/${PKG}" exec tsc --noEmit
pnpm --filter "./packages/${PKG}" test

git commit -am "refactor(${PKG}): migrate DocumentClient to lib-dynamodb (${SHARD})"
git push -u origin "ai/refactor-${SHARD}"
gh pr create --draft --base main \
  --title "refactor(${PKG}): DocumentClient migration (${SHARD})" \
  --body "Generated by batch run; type check and tests passed in CI."

Because of set -e, a type-check or test failure stops the script before anything is pushed. Send those shards to a reject queue rather than loosening types until they pass. Then review and merge in dependency order:

• One draft PR per package shard, small enough to read in one sitting.
• CODEOWNERS approval required; no auto-merge for generated changes.
• Merge leaf packages first and applications last, so each merge builds on reviewed code.
• Tag the pre-refactor commit so you can roll back the whole migration if a behavior change slips through.

Cost and scheduling

Batch inference is priced separately from on-demand inference; check the Amazon Bedrock pricing page for your model and Region rather than assuming a fixed discount. Your costs are driven by input tokens (the files you send) and output tokens (the files you get back), so the filtering in Step 1 matters more than any other optimization. A job that is scheduled but doesn’t start before its timeout ends as Expired, so pick a timeoutDurationInHours that fits your overnight window, and subscribe to EventBridge job state change events instead of polling.

Worked example: migrating DynamoDB clients in a monorepo

1. The sharder finds the old import in files across many packages and writes small shards of up to 20 files each, recorded against commit a1b2c3d.
2. At night, CI builds the JSONL, uploads it and submits a Converse-type job with a 24-hour timeout.
3. In the morning the job is PartiallyCompleted. The manifest shows a handful of record errors; those shards go straight to the reject queue.
4. Parsing rejects two more replies that weren’t valid JSON and one that edited a file outside its shard.
5. The remaining shards are applied in worktrees. Most pass type checks and tests and become draft PRs; the rest join the reject queue for an engineer to finish interactively.
6. Reviewers merge the shared packages first, then the services, over a couple of days.

Troubleshooting

Production checklist

• ☐ Only mechanical, testable transforms go to batch; judgment calls stay interactive
• ☐ Shards aligned with packages and recorded against one commit SHA
• ☐ Converse-format records with a strict “JSON only” reply contract and a “skipped” escape hatch
• ☐ Caller and service roles scoped to the model, job and S3 prefixes
• ☐ Results joined on recordId; invalid JSON and out-of-shard edits rejected
• ☐ Every shard type-checked and tested in a disposable worktree before any PR
• ☐ Draft PRs per package, CODEOWNERS approval, no auto-merge, rollback tag
• ☐ Batch jobs tagged and costed separately from interactive usage

FAQ

Can a batch record call tools, like reading another file?

No. Batch inference doesn’t support tool calling; each record is processed on its own. Put everything the model needs in the record.

Can I rely on structured output in batch jobs?

The AWS documentation is inconsistent on this: the structured outputs page lists batch inference as supported, while the batch inference guide says it isn’t. Until that’s settled, ask for JSON in the prompt and validate every reply yourself, as shown above.

Why full files instead of diffs?

Generated unified diffs often fail to apply because of off-by-one hunk headers. Full content plus git diff on your side is more reliable for small files. For very large files, shard by file and keep the output budget in mind.

Can I use prompt caching in batch?

No. Prompt caching applies to on-demand inference, not to the batch inference API.

Official documentation

• Process multiple prompts with batch inference
• Format and upload your batch inference data
• Required permissions for batch inference
• Create a custom service role for batch inference
• View the results of a batch inference job

Related on CheatCoders: Bedrock prompt caching and batch inference · AWS Batch for overnight agent jobs · Token budgets per PR and per engineer

Last updated on October 10, 2026