Day 26: Citations or It Didn’t Happen

Code RAG that “sounds right” without a verifiable path:line is a liability. Seniors treat citations like typed return values: if the span is not in the retrieval set, the answer is refused — not polished into a confident paragraph. Day 26 makes that policy enforceable in the gateway, the prompt, the UI, and the eval suite.

⚡ TL;DR: Require every claim to cite file:start-end drawn only from retrieved chunks. Validate path + quote against the retrieval payload before the response leaves your API. Fail closed on missing or hallucinated paths. Measure citation fidelity separately from prose quality.

Citation as a contract, not a vibe

Define a response schema the model must fill and your gateway must verify. Prefer structured outputs / tool-choice over free prose with “please cite.”

// ✅ Citation contract shared by model + gateway + UI
type Citation = {
  path: string;       // repo-relative
  startLine: number;
  endLine: number;
  quote: string;      // exact substring from retrieved text
};

type Answer = {
  summary: string;
  citations: Citation[];
  refuse?: { reason: "no_span_in_retrieval" | "ambiguous" | "stale_index" };
};

SYSTEM = "Please cite files when possible."

Pin additionalProperties: false on the JSON schema. Require minItems: 1 on citations unless refuse is set. Treat a missing refuse + empty citations as a hard server error, not a soft warning.

Fail closed against the retrieval set

Never trust the model’s memory of a path. Build a set of allowed (path, text, line window) from the retriever, then check every citation before streaming completes (or buffer then flush).

from dataclasses import dataclass

@dataclass(frozen=True)
class RetrievedChunk:
    path: str
    start: int
    end: int
    text: str

def cite_ok(cite: dict, chunks: list[RetrievedChunk]) -> bool:
    for c in chunks:
        if c.path != cite["path"]:
            continue
        if cite["endLine"] < c.start or cite["startLine"] > c.end:
            continue
        norm = c.text.replace("\r\n", "\n")
        q = cite["quote"].strip()
        if q and q in norm:
            return True
    return False

def gate(answer: dict, chunks: list[RetrievedChunk]) -> dict:
    if answer.get("refuse"):
        return answer
    cites = answer.get("citations") or []
    bad = [c for c in cites if not cite_ok(c, chunks)]
    if not cites or bad:
        return {
            "refuse": {"reason": "no_span_in_retrieval"},
            "summary": "",
            "citations": [],
            "debug": {"bad": bad, "retrieved": [c.path for c in chunks]},
        }
    return answer

Fuzzy matching (“close enough quotes”) is how hallucinations sneak back in. Allow only trivial whitespace normalization. If the model paraphrases, regenerate once with a stricter instruction; then refuse.

Prompt for quotes, not vibes

Put retrieved context in a delimited section and ban outside knowledge for path claims. Restate the rule every turn — models drift when the last user message is aggressive.

CONTEXT (authoritative; cite only from here):
<<<
path=services/api/src/auth.ts lines=10-40
...source...
>>>

Rules:
1. Every factual claim needs citations[].
2. quote must be copied verbatim from CONTEXT.
3. If CONTEXT lacks the answer, set refuse.reason=no_span_in_retrieval.
4. Never invent paths, symbols, or line numbers.

For multi-hop answers, require one citation per atomic claim, not a single decorative file at the bottom.

Eval and UI that make refusal useful

Offline: score citation fidelity (cited ⊆ retrieved) and span support (quote ∈ chunk) separately from helpfulness rubrics. Online: when the gate refuses, show top retrieved paths so developers can still jump into the repo. Log trace_id, retrieval ids, and gate outcomes for Day 29 failure taxonomy.

Production checklist

• [ ] JSON schema / tool output enforces citations[] or refuse
• [ ] Gateway validates path + quote against retrieval payload
• [ ] Eval suite tracks citation fidelity as a release gate
• [ ] UI renders clickable path:line and useful refuse states
• [ ] Logs store retrieval IDs for replay when citations fail
• [ ] Chaos test: strip chunks and assert refuse, not hallucination

Series navigation

← Day 25 · Day 27 →

Last updated September 11, 2026