Day 95: Incident Response When the Agent Goes Wrong

Agents will go wrong: bad tool call, poisoned retrieval, runaway loop. Incident response needs kill switches, replay, and customer-safe communication — practiced before SEV-1.

⚡ TL;DR: Global and per-feature kill switches. Freeze mutating tools first. Replay traces to find blast radius. Communicate without speculative AI blame language.

First 15 minutes

1. Flip AI mode to read_only or offline (Day 78).
2. Disable specific tool groups if known.
3. Snapshot gateway traces for the window.
4. Identify tenants/users touched.
5. Customer comms template — facts only.

# ✅ Kill switch read path in gateway
def guard(feature: str, flags: dict):
    if flags.get("ai_global_off"):
        raise Unavailable("AI temporarily disabled")
    if not flags.get(f"feature:{feature}", True):
        raise Unavailable(f"{feature} disabled")

Replay

Use Day 66 audit trails: same inputs → see tool calls. Prefer twin for re-runs.

❌ Re-running the failing mutate against prod to “see if it happens again.”

Failure modes

Kill switch that only stops new sessions while in-flight Step Functions keep mutating. Cancel running executions and revoke tool IAM with SCPs if needed. Comms that speculate about “AI bias” before facts — stick to impact and actions.

Closing checklist

• [ ] Kill switches tested quarterly
• [ ] On-call knows flag names
• [ ] Trace export path documented
• [ ] Comms templates ready
• [ ] Postmortem includes prompt/tool versions

Series navigation

Day 94: Change Management for Prompts · Day 96: Hiring and Team Topology for AI Platform

Last updated September 11, 2026