Day 78: Load Shedding When the Model Is Sick

Models get sick: regional throttles, elevated error rates, quality cliffs after a provider change. Your product still has users. Load shedding is how you stay honest — degrade or refuse with a useful error instead of hanging tools halfway through a mutate.

⚡ TL;DR: Detect provider pain with error-rate and latency burn. Shed: disable non-critical features, route to draft models, reject new agent loops, freeze mutating tools. Always return a typed error the UI can explain.

Signals that mean “sick”

# ✅ Sliding-window health for the model backend
from collections import deque

class ModelHealth:
    def __init__(self, window=200):
        self.events = deque(maxlen=window)

    def record(self, ok: bool, latency_ms: float) -> None:
        self.events.append((ok, latency_ms))

    def unhealthy(self) -> bool:
        if len(self.events) < 50:
            return False
        err = sum(1 for ok, _ in self.events if not ok) / len(self.events)
        p95 = sorted(l for _, l in self.events)[int(0.95 * len(self.events))]
        return err > 0.08 or p95 > 12_000

Pair with Bedrock ThrottlingException counts and multi-region failover (Day 38).

Shedding ladder

1. Stop speculative / nice-to-have agents (summarize-all-PRs bots).
2. Draft-then-verify off — serve draft-only with banner, or block.
3. Mutating tools → suggest-only (read tools stay up).
4. Reject new sessions with 503 model_degraded + retry-after.
5. Kill switch feature flag for the whole AI surface.

// ✅ Typed degradation the client can render
type AiMode = "full" | "read_only" | "draft_only" | "offline";

export function nextMode(h: { errorRate: number; throttled: boolean }): AiMode {
  if (h.errorRate > 0.2 || h.throttled) return "offline";
  if (h.errorRate > 0.08) return "read_only";
  if (h.errorRate > 0.03) return "draft_only";
  return "full";
}

❌ Retrying mutate tools in a tight loop while the provider is 500ing — that doubles damage when it recovers.

Failure modes

Flapping health (50% errors) causes mode thrash that confuses users. Hysteresis: enter read_only fast, leave only after N healthy minutes. Never shed by dropping requests without a typed body — mobile clients will retry mutates blindly.

Closing checklist

• [ ] Health signal from errors + latency + throttle codes
• [ ] Feature flags for AI mode: full / read_only / draft_only / offline
• [ ] Mutating tools blocked under degradation
• [ ] User-visible banner with status page link
• [ ] Runbook: who flips flags, who opens the incident

Series navigation

Day 77: Caching RAG Answers Safely · Day 79: FinOps Dashboards Engineers Open

Last updated September 11, 2026