Bedrock Multi-Region Failover: Survive Model Outages Without Broken UX

When us-east-1 Bedrock throttles or a model ID soft-fails, IDE assistants and PR bots should degrade — not spin forever or throw opaque 5xx pages. Multi-region failover is more than a second client config: prompts, tool schemas, inference profiles, and guardrails must be compatible, and the UX must advertise degraded mode so engineers trust the fallback.

⚡ TL;DR: Health-check primary model/region; fail over to a pinned secondary with prompt parity tests. Cap retries, show degraded banners, preserve idempotency keys. Prefer inference profiles / cross-region options where available, but still test tool and guardrail parity. See Bedrock throughput provisioning and Multi-model routing.

Explicit primary / secondary contracts

# bedrock/failover.py
from dataclasses import dataclass
import time

@dataclass(frozen=True)
class Endpoint:
    region: str
    model_id: str
    guardrail_id: str | None

PRIMARY = Endpoint("us-east-1", "anthropic.claude-sonnet-4-5", "gr-primary")
SECONDARY = Endpoint("us-west-2", "anthropic.claude-sonnet-4-5", "gr-secondary")

TRANSIENT = {429, 503, 529}

def converse_with_failover(client_for, payload: dict) -> dict:
    errors = []
    for ep in (PRIMARY, SECONDARY):
        client = client_for(ep.region)
        try:
            return client.converse(**{**payload, "modelId": ep.model_id,
                                      **({"guardrailConfig": {"guardrailIdentifier": ep.guardrail_id}} if ep.guardrail_id else {})})
        except Exception as e:
            code = getattr(e, "response", {}).get("ResponseMetadata", {}).get("HTTPStatusCode", 0)
            errors.append((ep.region, code, str(e)[:200]))
            if code not in TRANSIENT and code != 0:
                break  # ✅ don’t fail over on 4xx prompt errors
            time.sleep(0.2)
    raise RuntimeError(f"bedrock_unavailable:{errors}")

Prompt and tool parity gates

Failover that changes tool JSON shape mid-session is worse than an outage. CI must converse against both regions with the same system prompt hash, tool schema digest, and a canary task.

# ✅ CI canary
pnpm evals:canaries --region us-east-1 --region us-west-2 --prompt-hash $HASH
# ❌ Shipping a prompt only validated in one region

Degraded-mode UX

Preserve request idempotency keys across retries so tools that write (ticket drafts, comments) do not double-post — same discipline as Bedrock Agents idempotent tools.

Health checks and sticky recovery

Probe every 30s with a tiny Converse; mark primary healthy only after N successes. Avoid flapping: require 60s healthy before failing back. Emit metrics: bedrock_failover_total, bedrock_region, ttfb_ms.

Closing checklist

✅ Dos
– ✅ Pin secondary model IDs and guardrail IDs explicitly
– ✅ Canary both regions on every prompt/tool change
– ✅ Fail over on throttles/5xx; not on validation 4xx
– ✅ Show degraded UX; keep idempotency keys
– ✅ Hysteresis on failback to stop flapping

❌ Don’ts
– ❌ Don’t strip guardrails during failover
– ❌ Don’t splice streams across regions mid-response
– ❌ Don’t assume model IDs are identical everywhere without testing
– ❌ Don’t infinite-retry the primary during a regional event
– ❌ Don’t hide failover from on-call dashboards

Related reading

• Bedrock Throughput Provisioning: Avoid Throttles During Sev-1 Incidents
• Multi-Model Routing: Cheap Draft Models Plus Expensive Verifiers
• Bedrock Agents: Idempotent Tool Calls Against DynamoDB Writes
• Cross-Account Bedrock Access: Platform Teams Without Shared Keys

Last updated on September 11, 2026