8 views
This guide focuses on AI platform gateway for production systems, with practical trade-offs for reliability, security, and cost.
AI platform gateway: production guidance
Every squad rolling its own Bedrock client is how you get god keys, no budgets, and unreplayable incidents. Day 80 is the platform AI gateway: one internal API that owns auth, budget, routing, traces, and policy for every product squad.⚡ TL;DR: Front all model and tool calls with a gateway. Issue per-service credentials. Enforce budgets, model allowlists, prompt versions, and OTel. Product code never holds long-lived provider keys.
Architecture sketch
text
Squad service ── mTLS/JWT ──► AI Gateway ──► Bedrock / tools
│
├── budget + rate limits
├── prompt registry pin
├── router (draft/verify)
└── traces + cost dims
python
# ✅ Gateway request contract
from pydantic import BaseModel, Field
class GatewayRequest(BaseModel):
service: str
feature: str
prompt_id: str
prompt_version: str | None = None # default = prod pin
messages: list[dict]
tools: list[str] = Field(default_factory=list)
max_input_tokens: int = 32_000
stream: bool = True
Non-negotiables
| Control | Rule |
|---|---|
| Auth | Per-service IAM/JWT; no shared keys |
| Budget | Hard reject when squad monthly $ exhausted |
| Models | Allowlist; pin versions |
| Tools | Gateway mediates; OPA/Cedar on mutate |
| Observability | One trace id from edge to model |
typescript
❌ Letting product Lambdas call // ✅ Client SDK — thin, no credentials in app config
export async function complete(req: GatewayRequest) {
return fetch("https://ai-gateway.internal/v1/complete", {
method: "POST",
headers: { Authorization: `Bearer ${await svcToken()}` },
body: JSON.stringify(req),
});
}
bedrock-runtime directly with a wide account role.
Project acceptance criteria
- [ ] At least two squads migrated off direct Bedrock
- [ ] Budget reject path tested in staging
- [ ] Trace shows gateway → model → tool
- [ ] Prompt hot-edit in prod impossible (Day 94)
- [ ] Runbook for gateway degradation (Day 78)
Failure modes
Gateway as SPOF: multi-AZ, cached prompt pins, and a degrade mode that returns offline errors instead of hanging. Squads bypassing the gateway with direct Bedrock calls — detect via CloudTrail and page after grace period.Closing checklist
- [ ] Design review of trust boundaries
- [ ] SDK + migration guide published
- [ ] Cost dims mandatory
- [ ] Load test admission control
- [ ] Security review of tool mediation
Series navigation
Day 79: FinOps Dashboards Engineers Open · Day 81: Long-Term Memory Without Becoming Creepy Last updated September 11, 2026 Related: Day 90: Project: Incident Timeline Summarizer; Day 96: Hiring and Team Topology for AI Platform; Day 60: Project: Migration Strangler Assistant.Watch: 100 Days of AI on YouTube
Short videos from CheatCoders, one AI engineering topic per day.
Most viewed
- Python Decorators Explained: From Simple Wrappers to Production Patterns
- Claude Code vs Cursor vs GitHub Copilot: Choose by Workflow, and Set Up One Repo That Works With All Three
- AI Agent Frameworks in 2025: LangGraph vs CrewAI vs AutoGen vs Raw API
- Python String Methods: Every str Method With Real Production Examples
- Css Animation Performance: Performance-Optimized Motion for Modern UIs
Newly added
- AWS Systems Manager Parameter Store: Hierarchical Runtime Config for Coding Agents
- Amazon Cognito Identity Pools: Federate Coding-Agent Tool Callers Without Long-Lived Keys
- Amazon Athena for Coding Agents: Query Tool Audit Logs and Eval Traces on S3 Without a Warehouse
- AWS Batch for Coding Agents: Parallel Overnight PR Reviews and Long-Running Jobs Without Lambda Timeouts
- Amazon Bedrock Prompt Management: Version and A/B Test Coding-Agent System Prompts
Deep-dive PDF
Get the expanded guide for this post — extra diagrams-style checklists, failure modes, and a production walkthrough. Free when you subscribe to CheatCoders.
Already subscribed? or open the subscribe page.
Discover more from CheatCoders
Subscribe to get the latest posts sent to your email.
