Day 93: SLOs for AI Features

AI features need SLOs beyond HTTP 200. Availability, faithfulness, and latency compete — pick which ones page humans, and which only ticket.

⚡ TL;DR: Define SLIs: success rate, grounded-answer rate, p95 latency, budget burn. Page on availability + one quality signal. Don’t page on every eval dip.

Example SLO set

# ✅ Error budget for availability
def error_budget_remaining(uptime: float, slo=0.999, window_min=30*24*60):
    allowed_down = (1 - slo) * window_min
    used = (1 - uptime) * window_min
    return allowed_down - used

❌ Paging the on-call because offline NDCG moved 0.3 points overnight.

Failure modes

Faithfulness jobs sampling only happy-path FAQ questions. Include adversarial and out-of-domain. Latency SLOs that include waiting on humans will never be met — exclude gate waits from the interactive SLO.

Closing checklist

• [ ] SLIs documented per AI feature
• [ ] Page vs ticket explicitly chosen
• [ ] Faithfulness sampling job exists
• [ ] Latency excludes human-gate wait (Day 76)
• [ ] Review SLOs quarterly

Series navigation

Day 92: Threat Model an AI Feature · Day 94: Change Management for Prompts

Last updated September 11, 2026