RAG Evaluation on Bedrock: Offline Suites Before You Trust Code Answers

Codebase RAG demos beautifully on three cherry-picked questions and then confidently cites the wrong file in production. The unfair advantage is an offline evaluation suite: labeled questions, gold file paths / spans, frozen retrieval + generation configs, and CI gates that fail the deploy when hit-rate or citation fidelity regresses. Bedrock (Knowledge Bases or your own retrieve-then-generate on Anthropic/Titan) does not excuse you from measurement — it just changes which API you wrap.

⚡ TL;DR: Build a versioned JSONL pack of ≥50–100 real developer questions with gold paths[] (and optional line spans). Score retrieval recall@k and citation fidelity separately from answer prose quality. Run the suite on every index rebuild and prompt/model change. Block merge if recall@5 drops more than an agreed budget (e.g. 3 points) or if citation-required answers lack file paths. Combine with citation-required RAG and the store choice notes in OpenSearch vs Aurora pgvector.

What “good” means for code RAG (not chatbot vibes)

For internal coding assistants, prioritize:

1. Retrieval hit-rate — gold path appears in top-k chunks
2. Citation fidelity — model quotes paths that were actually retrieved (no hallucinated files)
3. Answer usefulness — human or LLM-judge rubric after 1 and 2 pass

Optimizing 3 alone is how you ship eloquent wrongness.

// eval/types.ts
export type GoldQuestion = {
  id: string;
  question: string;
  // ✅ Paths relative to repo root that must appear in retrieved set
  goldPaths: string[];
  // Optional: at least one chunk should cover these symbols
  goldSymbols?: string[];
  tags?: ("auth" | "infra" | "api" | "data")[];
  difficulty?: "easy" | "medium" | "hard";
};

export type RetrievalScore = {
  id: string;
  recallAtK: number; // fraction of goldPaths found in topK
  hit: boolean;      // recallAtK > 0 (or == 1 if you require all)
  retrievedPaths: string[];
};

export type GenerationScore = {
  id: string;
  citesOnlyRetrieved: boolean;
  citedPaths: string[];
  hallucinatedPaths: string[];
  answerChars: number;
};

Build the pack from real tickets, not synthetic trivia

Mine the last quarter of Slack/Jira/PR questions. Deduplicate. Force each item to name files a senior would open. Exclude questions that need live prod state.

{"id":"q-014","question":"Where do we verify Cognito JWT on the public API?","goldPaths":["services/api/src/middleware/auth.ts","packages/auth/src/cognito.ts"],"tags":["auth"],"difficulty":"easy"}
{"id":"q-015","question":"How does the orders worker handle partial SQS batch failure?","goldPaths":["services/orders/src/handlers/processBatch.ts"],"tags":["api"],"difficulty":"medium"}

Version the pack in git (eval/packs/code-rag-v3.jsonl). When the repo moves files, update gold paths in the same PR as the move — otherwise you fake a retrieval regression.

Retrieve with the same path you ship

Call Bedrock Knowledge Base Retrieve (or your OpenSearch/pgvector client) with the production index id, chunking, and hybrid settings. Offline eval that uses a laptop FAISS dump will not catch the chunking bug you just shipped.

// eval/retrieve-bedrock.ts
import {
  BedrockAgentRuntimeClient,
  RetrieveCommand,
} from "@aws-sdk/client-bedrock-agent-runtime";

const client = new BedrockAgentRuntimeClient({});

export async function retrievePaths(opts: {
  knowledgeBaseId: string;
  question: string;
  k: number;
}): Promise<string[]> {
  const out = await client.send(
    new RetrieveCommand({
      knowledgeBaseId: opts.knowledgeBaseId,
      retrievalQuery: { text: opts.question },
      retrievalConfiguration: {
        vectorSearchConfiguration: { numberOfResults: opts.k },
      },
    }),
  );

  const paths = new Set<string>();
  for (const r of out.retrievalResults ?? []) {
    const uri = r.location?.s3Location?.uri ?? r.metadata?.["x-amz-bedrock-kb-source-uri"];
    if (typeof uri === "string") {
      // normalize s3://bucket/prefix/repo/path → repo path
      paths.add(normalizeToRepoPath(uri));
    }
  }
  return [...paths];
}

function normalizeToRepoPath(uri: string): string {
  const marker = "/repo/";
  const i = uri.indexOf(marker);
  return i >= 0 ? uri.slice(i + marker.length) : uri;
}

export function recallAtK(gold: string[], retrieved: string[]): number {
  if (!gold.length) return 1;
  const set = new Set(retrieved);
  const hits = gold.filter((g) => set.has(g) || [...set].some((r) => r.endsWith(g)));
  return hits.length / gold.length;
}

Chunking strategy dominates recall — revisit Bedrock Knowledge Bases chunking style tradeoffs when recall@5 collapses after a reindex.

Score citations separately from fluency

Force the model to cite paths (system prompt + output schema). Then check every cited path against the retrieved set.

// eval/citations.ts
export function citationFidelity(opts: {
  retrievedPaths: string[];
  answerMarkdown: string;
}): { cited: string[]; hallucinated: string[]; citesOnlyRetrieved: boolean } {
  const cited = [
    ...opts.answerMarkdown.matchAll(/\b([a-zA-Z0-9._/-]+\.(?:ts|tsx|js|py|go|java))\b/g),
  ].map((m) => m[1]);

  const retrieved = new Set(opts.retrievedPaths);
  const hallucinated = cited.filter(
    (p) => ![...retrieved].some((r) => r === p || r.endsWith("/" + p) || r.endsWith(p)),
  );

  return {
    cited,
    hallucinated,
    citesOnlyRetrieved: hallucinated.length === 0 && cited.length > 0,
  };
}

Wire citation-required generation as in Citation-Required RAG Answers. An answer with zero citations fails the suite even if prose sounds right.

Gate deploys in CI

// eval/gate.ts
export type SuiteReport = {
  n: number;
  meanRecallAt5: number;
  pctHitAt5: number;
  pctCitationClean: number;
};

export function assertNoRegression(
  baseline: SuiteReport,
  candidate: SuiteReport,
  budget = { recallDrop: 0.03, citationDrop: 0.05 },
) {
  // ❌ Shipping because “answers feel better in playground”
  if (baseline.meanRecallAt5 - candidate.meanRecallAt5 > budget.recallDrop) {
    throw new Error(
      `recall_regression:${baseline.meanRecallAt5}->${candidate.meanRecallAt5}`,
    );
  }
  if (baseline.pctCitationClean - candidate.pctCitationClean > budget.citationDrop) {
    throw new Error("citation_fidelity_regression");
  }
  // ✅ Require minimum absolute floors too
  if (candidate.pctHitAt5 < 0.7) throw new Error("hit_rate_below_floor");
}

Run on:

• Index rebuild (new chunker, embed model, metadata filters)
• Prompt / generator model changes
• Retriever hybrid weight changes

Store baseline reports as artifacts (eval/baselines/code-rag-v3.json) committed or pulled from S3 with immutability.

Triage failures like test flakes

Bucket failures: wrong chunk size, missing metadata filter (lang/package), stale index, gold path moved, question ambiguous. Only mark gold as wrong after a human agrees — do not silently edit gold to match a bad index.

export type FailureBucket =
  | "retrieval_miss"
  | "stale_gold_path"
  | "ambiguous_question"
  | "citation_hallucination"
  | "generator_ignored_context";

Checklist

• [ ] Versioned JSONL pack with gold paths; ≥50 questions spanning auth/infra/api
• [ ] Offline retrieve uses prod KB/index settings
• [ ] Metrics: recall@k, hit@k, citation fidelity (separate from LLM-judge)
• [ ] Citation-required generation enabled for scored answers
• [ ] CI gate vs baseline with explicit drop budgets + absolute floors
• [ ] Reindex and prompt changes both trigger the suite
• [ ] Failure triage buckets; gold updates only via reviewed PRs
• [ ] Related store/chunking decisions documented with eval evidence

Related reading

• Citation-Required RAG Answers: Force Models to Quote File Paths
• RAG: OpenSearch vs Aurora pgvector for Codebase Chat
• Bedrock Agents: Tool Use, Memory, and Guardrails
• AI Code Review Bots: IAM, Secrets, and Least-Privilege Pipelines

Last updated on September 11, 2026