Citation-Required RAG Answers: Force Models to Quote File Paths

Ungrounded chat answers are how “the cache TTL is 5 minutes” ships into a wiki when the code says TTL_SECONDS = 30. For developer RAG, citations are not a nice-to-have—they are the product. Force structured file path + line citations, validate them against retrieved chunks, and refuse to render answers that invent paths.

⚡ TL;DR: Prompt for JSON answers with claims[] each bearing path + start_line/end_line; validate every citation is in the retrieved set; strip or regenerate on miss; show clickable IDE deep-links in the UI. Tie to RAG Evaluation and Hallucination Triage.

Contract: claims with paths

// citation_schema.ts
export type CitedClaim = {
  text: string;
  path: string;       // repo-relative, e.g. packages/orders/src/cache.ts
  start_line: number;
  end_line: number;
};

export type GroundedAnswer = {
  summary: string;
  claims: CitedClaim[];
  open_questions: string[];
};

export const ANSWER_JSON_SCHEMA = {
  type: "object",
  required: ["summary", "claims", "open_questions"],
  properties: {
    summary: { type: "string" },
    claims: {
      type: "array",
      minItems: 1,
      items: {
        type: "object",
        required: ["text", "path", "start_line", "end_line"],
        properties: {
          text: { type: "string" },
          path: { type: "string" },
          start_line: { type: "integer", minimum: 1 },
          end_line: { type: "integer", minimum: 1 },
        },
      },
    },
    open_questions: { type: "array", items: { type: "string" } },
  },
} as const;

System prompt fragment:

You MUST answer ONLY with JSON matching the schema.
Every factual claim MUST cite a path+lines from the RETRIEVED CHUNKS block.
If evidence is insufficient, put the gap in open_questions and omit the claim.
Never invent file paths.

Validate before the UI paints

function validateCitations(
  answer: GroundedAnswer,
  retrieved: { path: string; start_line: number; end_line: number }[],
): { ok: true } | { ok: false; violations: string[] } {
  const violations: string[] = [];
  for (const c of answer.claims) {
    const hit = retrieved.some(
      (r) =>
        r.path === c.path &&
        c.start_line >= r.start_line &&
        c.end_line <= r.end_line,
    );
    if (!hit) violations.push(`${c.path}:${c.start_line}-${c.end_line}`);
  }
  return violations.length ? { ok: false, violations } : { ok: true };
}

async function answerOrRefuse(question: string, retrieved: Chunk[]) {
  const raw = await bedrockConverseJson(question, retrieved, ANSWER_JSON_SCHEMA);
  const check = validateCitations(raw, retrieved);
  if (!check.ok) {
    // One constrained retry, then hard refuse
    const retry = await bedrockConverseJson(
      question,
      retrieved,
      ANSWER_JSON_SCHEMA,
      { repair: `Invalid citations: ${check.violations.join(", ")}. Cite only retrieved ranges.` },
    );
    const check2 = validateCitations(retry, retrieved);
    if (!check2.ok) {
      return {
        type: "refused" as const,
        reason: "ungrounded_citations",
        violations: check2.violations,
        retrieved_paths: [...new Set(retrieved.map((r) => r.path))],
      };
    }
    return { type: "ok" as const, answer: retry };
  }
  return { type: "ok" as const, answer: raw };
}

Do not render markdown that only looks cited (see auth.ts) without machine-checked ranges — seniors will still click, juniors will trust.

UI: make verification one click

• Deep-link vscode://file/... or your internal code browser with #Lstart-Lend.
• Show retrieved chunk panels beside the answer; highlight cited spans.
• Surface refuses as “I don’t have grounded evidence” plus the retrieved file list—never a confident empty summary.

Metrics that keep the bar honest

Feed citation precision into the same harness as RAG Evaluation on AWS.

Closing checklist

Dos
– Use structured outputs / JSON schema for claims + paths + lines
– Validate citations ⊆ retrieved ranges before render
– Retry once with repair hints; then refuse
– Deep-link to exact lines in the IDE / code browser
– Track citation precision as a release gate

Donts
– Do not accept prose-only “sources” without paths
– Do not allow citations to files never retrieved
– Do not hide refuses behind generic apologies
– Do not skip line ranges (“somewhere in orders”)
– Do not let streaming UIs paint claims before validation completes (buffer JSON)

Related reading

• RAG Evaluation on AWS
• Hallucination Triage for RAG
• Bedrock Retrieval Filters
• Query Rewriting for Code RAG

Last updated on September 11, 2026