Codebase RAG demos beautifully on three cherry-picked questions and then confidently cites the wrong file in production. The unfair advantage is an offline evaluation suite: labeled questions, gold file paths / spans, frozen retrieval + generation configs, and CI gates that fail the deploy when hit-rate or citation fidelity regresses. Bedrock (Knowledge Bases or your own retrieve-then-generate on Anthropic/Titan) does not excuse you from measurement — it just changes which API you wrap.
⚡ TL;DR: Build a versioned JSONL pack of ≥50–100 real developer questions with gold
paths[](and optional line spans). Score retrieval recall@k and citation fidelity separately from answer prose quality. Run the suite on every index rebuild and prompt/model change. Block merge if recall@5 drops more than an agreed budget (e.g. 3 points) or if citation-required answers lack file paths. Combine with citation-required RAG and the store choice notes in OpenSearch vs Aurora pgvector.
What “good” means for code RAG (not chatbot vibes)
For internal coding assistants, prioritize:
- Retrieval hit-rate — gold path appears in top-k chunks
- Citation fidelity — model quotes paths that were actually retrieved (no hallucinated files)
- Answer usefulness — human or LLM-judge rubric after 1 and 2 pass
Optimizing 3 alone is how you ship eloquent wrongness.
// eval/types.ts
export type GoldQuestion = {
id: string;
question: string;
// ✅ Paths relative to repo root that must appear in retrieved set
goldPaths: string[];
// Optional: at least one chunk should cover these symbols
goldSymbols?: string[];
tags?: ("auth" | "infra" | "api" | "data")[];
difficulty?: "easy" | "medium" | "hard";
};
export type RetrievalScore = {
id: string;
recallAtK: number; // fraction of goldPaths found in topK
hit: boolean; // recallAtK > 0 (or == 1 if you require all)
retrievedPaths: string[];
};
export type GenerationScore = {
id: string;
citesOnlyRetrieved: boolean;
citedPaths: string[];
hallucinatedPaths: string[];
answerChars: number;
};
Build the pack from real tickets, not synthetic trivia
Mine the last quarter of Slack/Jira/PR questions. Deduplicate. Force each item to name files a senior would open. Exclude questions that need live prod state.
{"id":"q-014","question":"Where do we verify Cognito JWT on the public API?","goldPaths":["services/api/src/middleware/auth.ts","packages/auth/src/cognito.ts"],"tags":["auth"],"difficulty":"easy"}
{"id":"q-015","question":"How does the orders worker handle partial SQS batch failure?","goldPaths":["services/orders/src/handlers/processBatch.ts"],"tags":["api"],"difficulty":"medium"}
Version the pack in git (eval/packs/code-rag-v3.jsonl). When the repo moves files, update gold paths in the same PR as the move — otherwise you fake a retrieval regression.
Retrieve with the same path you ship
Call Bedrock Knowledge Base Retrieve (or your OpenSearch/pgvector client) with the production index id, chunking, and hybrid settings. Offline eval that uses a laptop FAISS dump will not catch the chunking bug you just shipped.
// eval/retrieve-bedrock.ts
import {
BedrockAgentRuntimeClient,
RetrieveCommand,
} from "@aws-sdk/client-bedrock-agent-runtime";
const client = new BedrockAgentRuntimeClient({});
export async function retrievePaths(opts: {
knowledgeBaseId: string;
question: string;
k: number;
}): Promise<string[]> {
const out = await client.send(
new RetrieveCommand({
knowledgeBaseId: opts.knowledgeBaseId,
retrievalQuery: { text: opts.question },
retrievalConfiguration: {
vectorSearchConfiguration: { numberOfResults: opts.k },
},
}),
);
const paths = new Set<string>();
for (const r of out.retrievalResults ?? []) {
const uri = r.location?.s3Location?.uri ?? r.metadata?.["x-amz-bedrock-kb-source-uri"];
if (typeof uri === "string") {
// normalize s3://bucket/prefix/repo/path → repo path
paths.add(normalizeToRepoPath(uri));
}
}
return [...paths];
}
function normalizeToRepoPath(uri: string): string {
const marker = "/repo/";
const i = uri.indexOf(marker);
return i >= 0 ? uri.slice(i + marker.length) : uri;
}
export function recallAtK(gold: string[], retrieved: string[]): number {
if (!gold.length) return 1;
const set = new Set(retrieved);
const hits = gold.filter((g) => set.has(g) || [...set].some((r) => r.endsWith(g)));
return hits.length / gold.length;
}
Chunking strategy dominates recall — revisit Bedrock Knowledge Bases chunking style tradeoffs when recall@5 collapses after a reindex.
Score citations separately from fluency
Force the model to cite paths (system prompt + output schema). Then check every cited path against the retrieved set.
// eval/citations.ts
export function citationFidelity(opts: {
retrievedPaths: string[];
answerMarkdown: string;
}): { cited: string[]; hallucinated: string[]; citesOnlyRetrieved: boolean } {
const cited = [
...opts.answerMarkdown.matchAll(/\b([a-zA-Z0-9._/-]+\.(?:ts|tsx|js|py|go|java))\b/g),
].map((m) => m[1]);
const retrieved = new Set(opts.retrievedPaths);
const hallucinated = cited.filter(
(p) => ![...retrieved].some((r) => r === p || r.endsWith("/" + p) || r.endsWith(p)),
);
return {
cited,
hallucinated,
citesOnlyRetrieved: hallucinated.length === 0 && cited.length > 0,
};
}
Wire citation-required generation as in Citation-Required RAG Answers. An answer with zero citations fails the suite even if prose sounds right.
Gate deploys in CI
// eval/gate.ts
export type SuiteReport = {
n: number;
meanRecallAt5: number;
pctHitAt5: number;
pctCitationClean: number;
};
export function assertNoRegression(
baseline: SuiteReport,
candidate: SuiteReport,
budget = { recallDrop: 0.03, citationDrop: 0.05 },
) {
// ❌ Shipping because “answers feel better in playground”
if (baseline.meanRecallAt5 - candidate.meanRecallAt5 > budget.recallDrop) {
throw new Error(
`recall_regression:${baseline.meanRecallAt5}->${candidate.meanRecallAt5}`,
);
}
if (baseline.pctCitationClean - candidate.pctCitationClean > budget.citationDrop) {
throw new Error("citation_fidelity_regression");
}
// ✅ Require minimum absolute floors too
if (candidate.pctHitAt5 < 0.7) throw new Error("hit_rate_below_floor");
}
Run on:
- Index rebuild (new chunker, embed model, metadata filters)
- Prompt / generator model changes
- Retriever hybrid weight changes
Store baseline reports as artifacts (eval/baselines/code-rag-v3.json) committed or pulled from S3 with immutability.
Triage failures like test flakes
Bucket failures: wrong chunk size, missing metadata filter (lang/package), stale index, gold path moved, question ambiguous. Only mark gold as wrong after a human agrees — do not silently edit gold to match a bad index.
export type FailureBucket =
| "retrieval_miss"
| "stale_gold_path"
| "ambiguous_question"
| "citation_hallucination"
| "generator_ignored_context";
Checklist
- [ ] Versioned JSONL pack with gold paths; ≥50 questions spanning auth/infra/api
- [ ] Offline retrieve uses prod KB/index settings
- [ ] Metrics: recall@k, hit@k, citation fidelity (separate from LLM-judge)
- [ ] Citation-required generation enabled for scored answers
- [ ] CI gate vs baseline with explicit drop budgets + absolute floors
- [ ] Reindex and prompt changes both trigger the suite
- [ ] Failure triage buckets; gold updates only via reviewed PRs
- [ ] Related store/chunking decisions documented with eval evidence
Related reading
- Citation-Required RAG Answers: Force Models to Quote File Paths
- RAG: OpenSearch vs Aurora pgvector for Codebase Chat
- Bedrock Agents: Tool Use, Memory, and Guardrails
- AI Code Review Bots: IAM, Secrets, and Least-Privilege Pipelines
Last updated on September 11, 2026
Discover more from CheatCoders
Subscribe to get the latest posts sent to your email.
