Day 38: Multi-Region Model Failover

Model APIs fail by region. Fail over generation without a retrieval plan and you answer from an empty or stale brain. Day 38 separates generation failover from index strategy and keeps UX honest when degraded.

⚡ TL;DR: Health-checked regional clients for Converse. Prefer async replica indexes with lag SLOs over dual-write chaos. Sticky sessions for side effects. Refuse or degrade visibly rather than hallucinate.

Health-checked clients

REGIONS = ["us-east-1", "us-west-2"]
clients = {r: boto3.client("bedrock-runtime", region_name=r) for r in REGIONS}

def converse_failover(messages, model_id_by_region: dict):
    errors = []
    for r in REGIONS:
        if not health[r].ok:
            continue
        try:
            return clients[r].converse(modelId=model_id_by_region[r], messages=messages)
        except Exception as e:
            errors.append((r, e)); health[r].mark_bad()
    raise RuntimeError(f"all_regions_failed: {errors!r}")

Index strategies

For code RAG, async replica + refuse if lag > SLA beats split-brain writes. Include region in idempotency keys when side effects might cross regions (Day 35).

Production checklist

• [ ] Circuit breakers per region
• [ ] Documented RPO/RTO for index
• [ ] Idempotency keys region-safe
• [ ] Staging game-day disabling primary
• [ ] Status-page copy for degraded mode
• [ ] Citation fail-closed still on in degrade path

Series navigation

← Day 37 · Day 39 →

Last updated September 11, 2026