Day 80: Project: Platform AI Gateway

Lab / Project: Platform AI Gateway

Every squad rolling its own Bedrock client is how you get god keys, no budgets, and unreplayable incidents. Day 80 is the platform AI gateway: one internal API that owns auth, budget, routing, traces, and policy for every product squad.

⚡ TL;DR: Front all model and tool calls with a gateway. Issue per-service credentials. Enforce budgets, model allowlists, prompt versions, and OTel. Product code never holds long-lived provider keys.

Architecture sketch

Squad service ── mTLS/JWT ──► AI Gateway ──► Bedrock / tools
                                 │
                                 ├── budget + rate limits
                                 ├── prompt registry pin
                                 ├── router (draft/verify)
                                 └── traces + cost dims

# ✅ Gateway request contract
from pydantic import BaseModel, Field

class GatewayRequest(BaseModel):
    service: str
    feature: str
    prompt_id: str
    prompt_version: str | None = None  # default = prod pin
    messages: list[dict]
    tools: list[str] = Field(default_factory=list)
    max_input_tokens: int = 32_000
    stream: bool = True

Non-negotiables

// ✅ Client SDK — thin, no credentials in app config
export async function complete(req: GatewayRequest) {
  return fetch("https://ai-gateway.internal/v1/complete", {
    method: "POST",
    headers: { Authorization: `Bearer ${await svcToken()}` },
    body: JSON.stringify(req),
  });
}

❌ Letting product Lambdas call bedrock-runtime directly with a wide account role.

Project acceptance criteria

• [ ] At least two squads migrated off direct Bedrock
• [ ] Budget reject path tested in staging
• [ ] Trace shows gateway → model → tool
• [ ] Prompt hot-edit in prod impossible (Day 94)
• [ ] Runbook for gateway degradation (Day 78)

Failure modes

Gateway as SPOF: multi-AZ, cached prompt pins, and a degrade mode that returns offline errors instead of hanging. Squads bypassing the gateway with direct Bedrock calls — detect via CloudTrail and page after grace period.

Closing checklist

• [ ] Design review of trust boundaries
• [ ] SDK + migration guide published
• [ ] Cost dims mandatory
• [ ] Load test admission control
• [ ] Security review of tool mediation

Series navigation

Day 79: FinOps Dashboards Engineers Open · Day 81: Long-Term Memory Without Becoming Creepy

Last updated September 11, 2026