On-demand Bedrock feels infinite until Friday when every eval suite and your on-call copilot collide. Throttles are capacity planning, not “retry harder.” Day 37 partitions lanes, buys Provisioned Throughput where incidents demand it, and sheds batch before interactive.
⚡ TL;DR: Separate interactive vs batch model IDs/PT. Canary PT. Load-shed batch first when throttle burn rises. Monitor throttles as an SLO.
Partition demand
| Lane | Traffic | Capacity |
|---|---|---|
| Interactive / on-call | spiky, latency-sensitive | PT or dedicated |
| PR review bots | medium | on-demand + concurrency cap |
| Nightly eval | huge | batch + separate account/PT off-peak |
INTERACTIVE_MODEL = "arn:aws:bedrock:...:provisioned-model/oncall-pt"
BATCH_MODEL = "anthropic.claude-..."
def model_for(lane: str) -> str:
return INTERACTIVE_MODEL if lane == "interactive" else BATCH_MODEL
Pre-warm and shed
Run synthetic invokes against PT every minute. When throttle rate exceeds budget, set batch concurrency to zero before touching on-call routes. Document the shed order in the Sev runbook next to Day 38 failover.
Production checklist
- [ ] Separate model IDs / PT for interactive vs batch
- [ ] Dashboard: throttles by lane
- [ ] Synthetic canary on PT
- [ ] Shed runbook ordered
- [ ] Monthly cost review so PT stays intentional
- [ ] Game-day: induce throttle, prove shed works
Series navigation
Last updated September 11, 2026
Discover more from CheatCoders
Subscribe to get the latest posts sent to your email.
