Bedrock Throughput Provisioning: Avoid Throttles During Sev-1 Incidents

The worst time to discover Bedrock on-demand throttling is minute twelve of a sev-1, when every on-call is asking the incident copilot the same question. Provisioned throughput and explicit fallback models are reliability features, not cost optimization theater.

⚡ TL;DR: Put on-call and ChatOps assistants on provisioned throughput (or reserved capacity) sized to peak concurrent sev traffic. Define a fallback model ID with prompt compatibility checks. Cache identical runbook answers. Alarm on ThrottlingException before humans notice. Pair with LLM Incident Runbooks and Bedrock Prompt Caching and Batch Inference.

Size for incident concurrency, not average Tuesday

// capacity planning — illustrative
type Workload = {
  name: string;
  peakConcurrentSevSessions: number;
  avgInputTokens: number;
  avgOutputTokens: number;
  tokensPerSecondNeeded: number;
};

export function estimateModelUnits(w: Workload): number {
  // Rough: peak sessions * (in+out) / seconds-per-answer
  const tokensPerMin =
    w.peakConcurrentSevSessions * (w.avgInputTokens + w.avgOutputTokens) *
    (60 / 20); // assume ~20s per answer
  // Map to model units per vendor docs; keep 2x headroom for sev-1
  return Math.ceil((tokensPerMin / 10_000) * 2);
}

const onCallAssistant: Workload = {
  name: "sev-copilot",
  peakConcurrentSevSessions: 25,
  avgInputTokens: 6_000,
  avgOutputTokens: 1_200,
  tokensPerSecondNeeded: 0, // derived
};

✅ Separate inference profiles for IDE autocomplete (spiky, cheap model) vs on-call (provisioned, frontier).
❌ One shared on-demand quota for marketing chatbots and sev copilots.

Fallback chain that preserves prompt contracts

import {
  BedrockRuntimeClient,
  ConverseCommand,
  ThrottlingException,
} from "@aws-sdk/client-bedrock-runtime";

const PRIMARY = process.env.BEDROCK_PRIMARY_PROFILE!; // provisioned
const FALLBACK = process.env.BEDROCK_FALLBACK_MODEL!; // on-demand smaller

export async function converseResilient(input: {
  messages: any[];
  system: any[];
}) {
  const client = new BedrockRuntimeClient({});
  try {
    return await client.send(new ConverseCommand({
      modelId: PRIMARY,
      messages: input.messages,
      system: input.system,
    }));
  } catch (e) {
    if (!(e instanceof ThrottlingException)) throw e;
    metrics.increment("bedrock.throttle.primary");
    // ✅ Fallback only if prompt features intersect (tools, JSON mode)
    assertPromptCompatible(PRIMARY, FALLBACK);
    return await client.send(new ConverseCommand({
      modelId: FALLBACK,
      messages: input.messages,
      system: input.system,
      // optionally drop tools that fallback cannot support
    }));
  }
}

Document tool-choice differences with Bedrock Converse API: Tool Choice Modes.

Cache runbooks; do not re-generate identical answers

import { createHash } from "node:crypto";

function cacheKey(question: string, runbookSha: string) {
  return createHash("sha256")
    .update(normalize(question) + ":" + runbookSha)
    .digest("hex");
}

// During sev-1, 10 engineers ask "how do we failover payments redis?"
// Serve the cached grounded answer keyed by runbook SHA.

Prompt Management aliases help you pin versions — Bedrock Prompt Management.

Alarms that page before the throttle bites users

# CloudWatch — illustrative
AlarmName: bedrock-oncall-throttles
MetricName: ThrottlingException
Namespace: Bedrock/OnCallAssistant
Threshold: 5
Period: 60
EvaluationPeriods: 2
AlarmActions: [sns:sev-paging]

Also track p95 latency and provisioned utilization. Cross-region failover patterns belong in the same reliability pack as VPC private endpoints — AI Coding in VPC.

Closing checklist

• [ ] On-call / ChatOps path uses provisioned throughput or reserved capacity
• [ ] Peak sev concurrency modeled with ≥2x headroom
• [ ] Fallback model declared with prompt/tool compatibility checks
• [ ] Runbook answer cache keyed by normalized question + doc SHA
• [ ] Alarms on throttles and provisioned utilization
• [ ] IDE traffic isolated from sev inference profiles

Related reading

• LLM Incident Runbooks: Ground On-Call Answers in CloudWatch Signals
• Bedrock Prompt Caching and Batch Inference: Cut Latency and Cost
• LLM Cost Controls: Token Budgets Per PR and Per Engineer
• Bedrock Converse API: Tool Choice Modes for Production Coding Agents

Last updated on September 11, 2026