Bedrock Multi-Region Failover: Keep Coding Assistants Working When a Region or Model Struggles

Your IDE assistant and PR bot both call one model in one AWS Region. On a busy afternoon the calls start failing: first a few throttling errors, then 503s. The IDE spinner spins, the PR bot times out and posts nothing, and engineers decide the assistant is unreliable. A second Region was always “on the roadmap”.

Failover for generative AI is more than a second client with a different region_name. The fallback has to use a model that is actually available there, the same guardrails, a prompt and tool schema that behave the same way, and, if you use retrieval, an index that exists in that Region. And users should know when they’re in degraded mode. This guide covers the layers in order: what Amazon Bedrock can route for you, what your client must handle, and how to keep the experience honest. It merges our earlier bootcamp lesson on multi-Region model failover into this guide.

⚡ TL;DR

• Start with cross-Region inference profiles. Bedrock picks a Region within a geography (or globally) to process each request, with no extra routing charge. It’s the cheapest resilience you can add.
• Add client-side failover for what routing can’t cover: the source Region itself failing, a model or feature you can’t use through a profile, or a deliberate switch to a backup model.
• Classify errors: back off and retry on 429, 500 and 529; consider another Region on persistent 503; never fail over on validation or access errors.
• Guardrails are configured per Region: create matching guardrails in each Region and test prompt, tool and guardrail parity in CI.
• Use a circuit breaker with hysteresis, keep idempotency keys across Regions, never splice a stream across Regions, and show a “degraded” banner.

Who this is for: teams running coding assistants or agents on Bedrock in production who need them to keep working through regional capacity problems. Examples use Python and boto3.

Layer 1: Let Bedrock route with cross-Region inference

An inference profile defines a model and the Regions requests can be routed to. With a geographic cross-Region inference profile (for example, a US or EU profile), Bedrock automatically selects a commercial Region within that geography to process your request. With a global profile, it can select any supported commercial Region. You call the profile ID or ARN as the modelId, and nothing else in your code changes.

Facts from the documentation that matter for planning:

• There’s no additional routing cost. The price is based on the Region you call the profile from.
• Data stays on the AWS network and is encrypted in transit between Regions. Your prompts and outputs may be processed outside your source Region, within the profile’s geography for geographic profiles.
• CloudTrail logs the request in your source Region; the additionalEventData.inferenceRegion field shows where it was processed.
• Your IAM policies must allow the model in the source Region and in every destination Region of the profile. If SCPs block unused Regions, allow the destination Regions or add the documented inference-profile exception. Global profiles need SCPs to allow "aws:RequestedRegion": "unspecified".
• Inference profiles don’t support Provisioned Throughput.

Choose geographic profiles when you have data residency requirements and global profiles when you don’t. For many teams, switching from a single-Region model ID to a geographic profile removes most capacity-related errors without any failover code.

Layer 2: Classify errors before you fail over

Failing over on every exception is how a single malformed prompt turns into a two-Region outage. Bedrock’s troubleshooting guide distinguishes the errors clearly:

One caveat on ResourceNotFoundException: if the model ID is simply not offered in a Region, that’s a configuration bug to fix before you need failover, not something to discover during an incident. Keep a per-Region model map and check it in CI with ListFoundationModels or by calling each configured profile.

import random
import time
from dataclasses import dataclass

import boto3
from botocore.config import Config
from botocore.exceptions import ClientError

RETRY_SAME_REGION = {"ThrottlingException", "InternalServerException", "ModelNotReadyException"}
FAIL_OVER = {"ServiceUnavailableException"}
OVERLOADED_HTTP = 529


@dataclass(frozen=True)
class Target:
    region: str
    model_id: str            # inference profile ID/ARN valid in this Region
    guardrail_id: str        # guardrails are per Region
    guardrail_version: str


@dataclass
class Breaker:
    failures: int = 0
    open_until: float = 0.0

    def available(self) -> bool:
        return time.time() >= self.open_until

    def record_failure(self, threshold: int = 5, cool_off: float = 60.0) -> None:
        self.failures += 1
        if self.failures >= threshold:
            self.open_until = time.time() + cool_off
            self.failures = 0

    def record_success(self) -> None:
        self.failures = 0


class FailoverClient:
    def __init__(self, targets: list):
        cfg = Config(retries={"max_attempts": 3, "mode": "adaptive"}, tcp_keepalive=True)
        self.targets = targets
        self.clients = {t.region: boto3.client("bedrock-runtime", region_name=t.region, config=cfg)
                        for t in targets}
        self.breakers = {t.region: Breaker() for t in targets}

    def converse(self, request: dict) -> tuple:
        """Returns (response, region_used). request excludes modelId and guardrailConfig."""
        errors = []
        for t in self.targets:
            if not self.breakers[t.region].available():
                errors.append((t.region, "breaker_open"))
                continue
            try:
                resp = self.clients[t.region].converse(
                    modelId=t.model_id,
                    guardrailConfig={"guardrailIdentifier": t.guardrail_id,
                                     "guardrailVersion": t.guardrail_version},
                    **request,
                )
                self.breakers[t.region].record_success()
                return resp, t.region
            except ClientError as e:
                code = e.response["Error"]["Code"]
                http = e.response["ResponseMetadata"].get("HTTPStatusCode", 0)
                errors.append((t.region, code))
                if code in FAIL_OVER or http == OVERLOADED_HTTP or code in RETRY_SAME_REGION:
                    # SDK retries already ran inside this call; count it against the Region.
                    self.breakers[t.region].record_failure()
                    time.sleep(random.uniform(0.1, 0.5))  # jitter before trying the next Region
                    continue
                raise  # validation, access, not found: same result everywhere
        raise RuntimeError(f"all Regions unavailable: {errors}")

The SDK’s own retry mode handles short bursts inside each call, so the loop only moves on after those retries are exhausted. The circuit breaker stops sending traffic to a Region that keeps failing, and the cool-off period prevents flapping: a Region must stay quiet for a while before it gets traffic again. In production you’d also probe an open Region with a tiny request and close the breaker only after several consecutive successes.

Notice tcp_keepalive=True. Bedrock’s troubleshooting guide notes that NAT gateways and interface VPC endpoints drop connections idle for more than 350 seconds; a pooled client in a quiet secondary Region is exactly the kind of connection that gets silently dropped. The OS keep-alive interval must also be below that timeout for the setting to help.

Layer 3a: Keep guardrails, prompts and tools in parity

A failover that changes behavior is worse than an error, because nobody notices. Three things must match across Regions:

• Guardrails. A guardrail’s configuration is stored in the Region where you created it, so the secondary Region needs its own guardrail with the same policies and its own ID and version. Never drop guardrails “to make failover work”. (Separately, guardrails can use cross-Region inference through a guardrail profile for their own evaluation capacity; that doesn’t remove the need for a guardrail in each source Region you call from.)
• Prompts and tool schemas. Use the same prompt text and the same tool definitions in both Regions, identified by a hash.
• Model behavior. If the secondary uses a different model or version, test it against the same canary tasks. Tool calling, forced tool choice and structured output support differ between models.

import hashlib
import json


def digest(obj) -> str:
    return hashlib.sha256(json.dumps(obj, sort_keys=True).encode()).hexdigest()[:12]


def parity_canary(client: "FailoverClient", system: list, tools: list, cases: list) -> None:
    """Run in CI on every prompt/tool/guardrail change, against each Region directly."""
    prompt_hash, tools_hash = digest(system), digest(tools)
    for t in client.targets:
        bedrock = client.clients[t.region]
        for case in cases:
            resp = bedrock.converse(
                modelId=t.model_id,
                system=system,
                messages=[{"role": "user", "content": [{"text": case["prompt"]}]}],
                toolConfig={"tools": tools},
                guardrailConfig={"guardrailIdentifier": t.guardrail_id,
                                 "guardrailVersion": t.guardrail_version},
                inferenceConfig={"maxTokens": 1024, "temperature": 0},
            )
            used = [b["toolUse"]["name"] for b in resp["output"]["message"]["content"]
                    if "toolUse" in b]
            assert case["expect_tool"] in used, (
                f"{t.region} prompt={prompt_hash} tools={tools_hash}: "
                f"expected {case['expect_tool']}, got {used or resp['stopReason']}")

Layer 3b: Decide what happens to retrieval

Generation is only half of a RAG assistant. Knowledge bases and vector stores are regional resources, so “fail over the model” can mean “answer without your code index”. Pick one strategy deliberately and write it down:

For code search, an asynchronous replica with a lag check is usually the sweet spot: re-index from the same Git commits in both Regions and record the indexed commit SHA, so the assistant can say which commit its answer is based on. Whatever you choose, keep citation checks on in degraded mode. An assistant that hallucinates because retrieval failed silently is worse than one that says “I can’t see the repository right now”.

Layer 3c: Honest degraded UX

Carry the same idempotency key through every attempt, in every Region. If a tool that posts a PR comment ran in the primary just before the failure, the retried call in the secondary must find that key and skip the duplicate post.

That only works if both Regions check the same ledger. Our earlier lesson suggested putting the Region into the idempotency key; that’s a correction worth making, because a Region-specific key gives the retry in the secondary a new key, so the duplicate PR comment gets posted after all. Keep the key Region-independent and do one of two things: keep the side-effect ledger in one place that both Regions’ tool runners can reach, or make sessions sticky, so a session’s side-effecting tools keep running where the session started while only model calls fail over. Read-only tools can run anywhere.

Worked example: a capacity problem on a Tuesday afternoon

1. The assistant uses a US geographic inference profile from us-east-1 as primary and a profile called from us-west-2 as secondary, each with its own guardrail of the same version number.
2. Throttling errors rise. The SDK’s adaptive retries absorb most of them; the rest trip the us-east-1 breaker after five failures and traffic moves to us-west-2. The IDE shows the backup-mode banner.
3. A user’s request with an oversized attachment fails with a validation error. It isn’t failed over; the user gets a clear error.
4. Retrieval uses the replica index, which is one commit behind; answers cite that commit SHA.
5. After the cool-off, probes succeed several times in a row, the breaker closes and traffic returns to us-east-1. The banner disappears.

Troubleshooting

Production checklist

• ☐ Cross-Region inference profiles in use; IAM and SCPs allow all destination Regions
• ☐ Per-Region map of model or profile IDs and guardrail IDs and versions, verified in CI
• ☐ Error classification: retry, fail over, or raise; no failover on 4xx
• ☐ Circuit breaker per Region with cool-off and probe-based recovery
• ☐ Parity canary across Regions on every prompt, tool or guardrail change
• ☐ Documented retrieval strategy with RPO/RTO targets for the index, lag monitoring or explicit refusal
• ☐ Status-page and in-app wording for degraded mode written before you need it
• ☐ Degraded-mode banner, no stream splicing, idempotency keys reused across Regions
• ☐ Game day: disable the primary in staging and watch the whole path

FAQ

Isn’t a cross-Region inference profile enough?

For capacity problems, often yes. Client-side failover covers cases routing can’t: problems with your source Region’s endpoint, models you can’t use through a profile, or a planned switch to a backup model.

Can I use Provisioned Throughput with failover?

Not through inference profiles, which don’t support it. Provisioned Throughput is tied to a Region, so a provisioned primary needs its own fallback plan, such as an on-demand profile in another Region.

Should the secondary use the same model?

Ideally yes, so behavior matches. If it can’t, treat the backup model as a separate release: run the same canaries and tool-calling tests before it’s allowed to take traffic.

Where do I see which Region processed a request?

In CloudTrail in your source Region, in the additionalEventData.inferenceRegion field.

Official documentation

• Increase throughput with cross-Region inference
• Geographic cross-Region inference (IAM and SCP requirements)
• Troubleshooting Amazon Bedrock API error codes
• Distribute guardrail inference across AWS Regions
• Inference profiles

Related on CheatCoders: Bedrock Knowledge Bases for monorepos · Idempotent tool calls with DynamoDB · PrivateLink for Bedrock

Last updated on October 10, 2026