Day 31: Lambda + Bedrock: Sync, Stream, and Batch

One Bedrock call pattern does not fit every client. Chat UIs need token streams; internal jobs need batch; mobile slash-commands may need buffered sync with tight deadlines. Hiding ThrottlingException behind infinite retries turns a capacity problem into a Sev-1 that looks like “AI is down.”

⚡ TL;DR: Map each route to sync, stream, or batch explicitly. Propagate throttles as 429s with metrics. Reserve interactive capacity (Day 37). Never run 10k eval prompts through the chat Lambda.

Pick the mode by deadline

Document the mapping in the service README. On-call should know which lane is burning quota.

Sync path with visible throttles

import boto3, botocore, time, random
bedrock = boto3.client("bedrock-runtime")

def converse_sync(model_id: str, messages: list, timeout_s: float = 20.0):
    deadline = time.time() + timeout_s
    attempt = 0
    while True:
        try:
            return bedrock.converse(modelId=model_id, messages=messages)
        except botocore.exceptions.ClientError as e:
            code = e.response["Error"]["Code"]
            if code not in {"ThrottlingException", "ModelTimeoutException"}:
                raise
            if time.time() >= deadline or attempt >= 3:
                raise  # caller → HTTP 429
            time.sleep(min(2 ** attempt + random.random(), max(0, deadline - time.time())))
            attempt += 1

while True:
    try:
        return bedrock.converse(...)
    except Exception:
        time.sleep(1)

Streaming without lying about latency

Lambda response streaming or Function URLs let you flush tokens, but you still own the full budget: retrieval + first token + completion. Emit time_to_first_token and cancel server-side work if the client disconnects.

Batch for eval and backfill

Do not fan out 10k sync Lambdas for offline packs. Use Bedrock batch jobs or SQS with concurrency caps and write results to S3. Keep interactive and batch on different model IDs / PT lanes (Day 37).

Production checklist

• [ ] Invoke mode documented per API route
• [ ] Throttles → 429 + Retry-After, not 500
• [ ] TTFT and total latency metrics by lane
• [ ] Batch path for eval / backfill
• [ ] Load test includes quota cliffs
• [ ] Runbook links to Day 37 / Day 38 failover

Series navigation

← Day 30 · Day 32 →

Last updated September 11, 2026