Day 29: Failure Taxonomy: Wrong File vs Wrong Advice

All RAG failures are not equal. Retrieving the wrong file needs a different fix than retrieving the right file and generating bad advice. If dashboards only show thumbs-down rate, you cannot improve the system — you can only argue about vibes. Day 29 installs a failure taxonomy shared by offline eval and production telemetry.

⚡ TL;DR: Label failures as retrieval_miss, citation_hallucination, stale_index, bad_generation, refusal_correct, tool_error. Route retries and pages by class. Automate labels from the citation gate and freshness markers whenever possible.

The classes that matter

1. retrieval_miss — gold path absent from top-k
2. citation_hallucination — cites path not in retrieval
3. stale_index — right path, wrong revision vs freshness SLO
4. bad_generation — right evidence, wrong conclusion
5. refusal_correct — appropriately said unknown
6. tool_error — timeout / throttle / IAM

from enum import Enum
from pydantic import BaseModel

class FailureClass(str, Enum):
    retrieval_miss = "retrieval_miss"
    citation_hallucination = "citation_hallucination"
    stale_index = "stale_index"
    bad_generation = "bad_generation"
    refusal_correct = "refusal_correct"
    tool_error = "tool_error"

class FailureEvent(BaseModel):
    request_id: str
    cls: FailureClass
    gold_paths: list[str] | None = None
    retrieved_paths: list[str]
    index_sha: str | None = None
    main_sha: str | None = None

// ❌ One bucket to rule them all
type Feedback = { thumbs: "down" }; // useless for routing

Different retries, different owners

Automate what you can

When Day 26’s gateway rejects citations, emit citation_hallucination automatically. When index_sha lags main beyond SLO, emit stale_index. Ask humans mainly for suspected bad_generation. Rising refusal_correct after tightening gates is often a success — do not page anyone for it.

Wire classes into Day 39 traces as span attributes (failure.cls) so a single trace_id explains the miss.

Production checklist

• [ ] FailureClass enum shared across eval + prod
• [ ] Dashboards split rates by class
• [ ] Runbooks per class with owners
• [ ] Weekly review samples 20 events per class
• [ ] CI fails on retrieval_miss regression over budget
• [ ] Thumbs-down UI optional secondary signal only

Series navigation

← Day 28 · Day 30 →

Last updated September 11, 2026