Day 47: Serving Custom Models Behind the Same API

Clients should not care whether a completion came from Claude, Titan, or your LoRA student. A router owns model ids, shadow traffic, and rollback. Day 47 defines the stable /v1/complete contract.

⚡ TL;DR: One API keyed by task. Map tasks → primary/shadow models in config. Shadow candidates; promote on eval + online metrics. Rollback via flag in minutes.

ROUTES = {
  "chat_general": {"primary": "claude", "shadow": None},
  "sdk_autocomplete": {"primary": "student_sdk_v3", "shadow": "claude"},
  "oncall_rag": {"primary": "claude", "shadow": None},
}

def complete(task: str, payload: dict):
    r = ROUTES[task]
    primary = invoke(r["primary"], payload)
    if r.get("shadow"):
        async_invoke(r["shadow"], payload)
    return primary

// ❌ Clients hardcode model IDs

Keep JSON schemas stable; hide model quirks in adapters. Version the HTTP API separately from model versions. Keep warm capacity for fallbacks (Day 37).

Production checklist

• [ ] Task→model map in config
• [ ] Shadow traffic for candidates
• [ ] Unified errors/timeouts
• [ ] Per-route dashboards
• [ ] Rollback < 5 minutes
• [ ] Clients never see raw model ids

Series navigation

← Day 46 · Day 48 →

Last updated September 11, 2026