Bedrock Prompt Caching and Batch Inference: Cut Latency and Cost

If your agent sends the same 4–8k token system prompt, tool schema, and style guide on every turn, you are paying full input price to re-digest work the model already did. If your nightly eval suite calls InvokeModel in a tight loop, you are paying on-demand rates for work nobody is watching. Amazon Bedrock gives you two different levers — prompt caching and batch inference — and mixing them up is an expensive mistake.

⚡ TL;DR: Prompt caching stores a stable prompt prefix so cache reads bill at a steep discount (AWS cites up to ~90% input cost reduction and up to ~85% latency improvement on supported models) when hit rates are high. Batch inference runs JSONL jobs from S3 at roughly ~50% of on-demand token price for latency-tolerant workloads. They are mutually exclusive on Bedrock — pick caching for interactive reuse, batch for deferred volume. Instrument cache write/read tokens; place cachePoint after the truly stable prefix; keep dynamic RAG/user turns after the point.

Prompt caching: make the prefix pay once

Caching helps when a long prefix repeats across calls: system instructions, many-shot examples, large tool JSON schemas, static policy docs. It does not discount output tokens. It does not help if every request mutates the first 90% of the prompt.

# invoke_with_cache.py — Messages API style with cache points (illustrative Claude-on-Bedrock)
import boto3, json

brt = boto3.client("bedrock-runtime")

SYSTEM = [
    {
        "text": OPEN_SYSTEM_PROMPT,  # long, stable — policies, tone, output contract
    },
    {
        # ✅ cachePoint AFTER stable content
        "cachePoint": {"type": "default"},
    },
]

TOOLS = [
    # tool defs can be large — keep them in the cached prefix when they rarely change
]

def chat(user_text: str, history: list[dict]):
    resp = brt.converse(
        modelId="anthropic.claude-3-7-sonnet-...",  # use a caching-supported ID in your region
        system=SYSTEM,
        toolConfig={"tools": TOOLS} if TOOLS else None,
        messages=history + [{"role": "user", "content": [{"text": user_text}]}],
        inferenceConfig={"maxTokens": 1024, "temperature": 0.2},
    )
    usage = resp.get("usage", {})
    # Inspect cache metrics when present — names vary slightly by API; log raw usage
    print(json.dumps(usage, default=str))
    return resp

For raw invoke_model with Anthropic-style bodies, the idea is the same: mark a cache boundary so the bytes before it are eligible for cache write/read.

{
  "system": [
    { "type": "text", "text": "…long stable system…" },
    { "type": "text", "text": "…tool schemas…", "cache_control": { "type": "ephemeral" } }
  ],
  "messages": [
    { "role": "user", "content": "dynamic question goes here" }
  ]
}

Placement rules that matter:

1. Stable → cache boundary → dynamic. User text, retrieved RAG chunks that change every call, and timestamps belong after the cache point.
2. Minimum size / TTL — models enforce minimum cacheable token counts and TTLs (often measured in minutes). A prefix used twice an hour may miss; a busy agent turnstream hits constantly.
3. Account-scoped — cache is not a global CDN of everyone’s prompts; design for your traffic pattern.

Illustrative napkin math (replace with your region’s Bedrock pricing page):

Prefix P = 6,000 tokens, reused N=1,000 times/day
Without cache: 6,000 * 1,000 = 6.0M input tokens/day billed at full input rate
With cache: 1 write + 999 reads at ~0.1× input (order-of-magnitude)
If write ≈ 1.25× and read ≈ 0.1× (illustrative ratios):
  effective ≈ 6k*1.25 + 6k*999*0.1  vs  6k*1000
  → large savings when N is high; negligible when N=2

Latency: skipping recomputation of a multi-thousand-token prefix often drops TTFT dramatically on supported models — AWS quotes up to ~85%. Measure TimeToFirstToken in your own traces before celebrating.

What not to put in the cached prefix

✅ System policy, output schema, static few-shots, tool JSON, product glossary
❌ Per-user PII, per-request RAG chunks, “current time is …”, rotating secrets
❌ Anything that forces the prefix to change every call (busts the cache)

For RAG: cache the instructions + tool schemas; retrieve docs after the boundary. If you have a large static corpus snippet shared by all users (e.g. public API reference), that can live in the cached prefix — tenant-private code cannot.

Batch inference: half price if nobody is waiting

Batch is an S3-in / S3-out job. You submit JSONL records, Bedrock runs when capacity allows, results land in your bucket. Pricing is typically ~50% of on-demand input/output token rates (confirm current Bedrock pricing). Completion can take minutes to hours — design async UX.

# 1) Build JSONL — each line is one model invocation record (shape per model)
# input.jsonl (illustrative)
# {"recordId":"eval-0001","modelInput":{ ... }}
# {"recordId":"eval-0002","modelInput":{ ... }}

aws s3 cp input.jsonl s3://bedrock-batch-jobs/prod/eval-2026-09-10/input.jsonl

aws bedrock create-model-invocation-job \
  --job-name eval-2026-09-10 \
  --role-arn arn:aws:iam::123456789012:role/BedrockBatchRole \
  --model-id anthropic.claude-3-5-haiku-... \
  --input-data-config '{"s3InputDataConfig":{"s3Uri":"s3://bedrock-batch-jobs/prod/eval-2026-09-10/input.jsonl"}}' \
  --output-data-config '{"s3OutputDataConfig":{"s3Uri":"s3://bedrock-batch-jobs/prod/eval-2026-09-10/out/"}}'

# poll_job.py — bounded poll; never block a user request thread on this
import boto3, time

bedrock = boto3.client("bedrock")

def wait_for_job(job_arn: str, budget_s: int = 3600):
    start = time.time()
    while time.time() - start < budget_s:
        job = bedrock.get_model_invocation_job(jobIdentifier=job_arn)
        status = job["status"]
        if status in ("Completed", "PartiallyCompleted", "Failed", "Stopped"):
            return job
        time.sleep(30)
    raise TimeoutError("batch_job_budget_exceeded")

Good batch workloads: nightly evals, offline classification/enrichment, doc summarization backfills, synthetic data generation.
Bad batch workloads: anything a human is staring at; multi-turn agents that need the previous answer now; “we got throttled on-demand so shove interactive traffic into batch.”

IAM for the batch role needs s3:GetObject on input, s3:PutObject on output, and bedrock:InvokeModel (plus PassRole from the caller). Encrypt the buckets; don’t leave completion JSONL public.

Caching vs batch — choose explicitly

❌ Don’t expect cache discounts inside a batch job.
❌ Don’t use batch as a throttle workaround for a broken on-demand capacity plan.
✅ Split pipelines at the product layer: “answer now” vs “run tonight.”

Observability and cost controls

# Log usage every call — illustrative fields
def log_usage(route: str, usage: dict):
    metrics = {
        "route": route,
        "inputTokens": usage.get("inputTokens"),
        "outputTokens": usage.get("outputTokens"),
        "cacheReadInputTokens": usage.get("cacheReadInputTokens"),
        "cacheWriteInputTokens": usage.get("cacheWriteInputTokens"),
    }
    # Emit to EMF / OTel — alert when cacheRead / (cacheRead+input) drops below target
    print(metrics)

Illustrative alerts:
– Cache hit ratio on agent system prefix < 70% during business hours → someone moved dynamic content above the cache point.
– On-demand token spend on the eval IAM role > $X/day → job not actually using batch.
– p99 TTFT regression after prompt edit → verify model still supports caching and prefix length.

Combine with inference profiles / application inference profiles when you need cross-region routing and consolidated cost attribution — still apply caching on the models that support it.

Closing checklist

✅ Dos
– ✅ Put cachePoint / cache_control after stable system + tools; keep user/RAG dynamic after
– ✅ Log cache read/write tokens and TTFT per route
– ✅ Send offline/eval volume through Batch Inference JSONL on S3
– ✅ Treat PartiallyCompleted as a real state — inspect per-record errors
– ✅ Re-validate pricing and supported model IDs in your region before forecasting savings

❌ Don’ts
– ❌ Don’t assume caching + batch stack on the same request path
– ❌ Don’t put per-request RAG or PII above the cache boundary
– ❌ Don’t block API Gateway/Lambda request threads waiting hours for a batch job
– ❌ Don’t forecast 90% savings with a 2-call-per-day prefix
– ❌ Don’t ignore output tokens — caching never discounts them

Related reading

• AWS Lambda Best Practices: Write Functions That Scale and Never Time Out
• Hyper-Scale Serverless API Platform on AWS
• Amazon Bedrock Agents: Tool Use, Memory, and Production Guardrails (companion post)
• RAG on AWS: OpenSearch vs Aurora pgvector for Codebase Chat (companion post)
• Official: Amazon Bedrock prompt caching and batch inference docs / pricing pages

Last updated on September 10, 2026