Code RAG Rerankers: Cross-Encoders After Cheap Vector Recall

Vector recall is cheap and wrong often enough to matter. Embedding neighborhood proximity ≠ “this snippet answers the question.” The unfair stack for code RAG: broad kNN recall (50–100 hits), then a cross-encoder reranker that scores (query, chunk) jointly so the final top-8 actually compile into grounded answers. Skip rerank and you will keep citing the wrong overload of authenticate.

⚡ TL;DR: Recall with vectors (and BM25 if hybrid); rerank with a cross-encoder or Bedrock rerank model; evaluate nDCG@8 and answer faithfulness before/after. Cap rerank batch size for p95. Pair with Bedrock chunking strategies and RAG evaluation harnesses.

Two-stage retrieval that seniors actually ship

// rerank_pipeline.ts
type Hit = { id: string; text: string; vectorScore: number };

export async function retrieveCode(query: string): Promise<Hit[]> {
  const recalled = await knnSearch(query, { k: 80 }); // ✅ broad recall
  // ❌ return recalled.slice(0, 8) — proximity ≠ relevance

  const reranked = await crossEncodeRerank(query, recalled, { topN: 8 });
  return reranked;
}

async function crossEncodeRerank(query: string, hits: Hit[], opts: { topN: number }) {
  // Batch pairs; score with cross-encoder hosted on SageMaker or Bedrock Rerank
  const scores = await rerankModel.score(
    hits.map((h) => ({ query, document: h.text }))
  );
  return hits
    .map((h, i) => ({ ...h, rerank: scores[i] }))
    .sort((a, b) => b.rerank - a.rerank)
    .slice(0, opts.topN);
}

✅ Always measure recall@80 separately from nDCG@8 after rerank.
❌ Tuning only k on the vector index and calling it “better RAG.”

Features that help code cross-encoders

Cross-encoders see the pair; help them with structure:

# format_pair.py
def format_chunk(meta: dict, body: str) -> str:
    return (
        f"FILE: {meta['path']}\n"
        f"SYMBOL: {meta.get('symbol', '')}\n"
        f"LANG: {meta['language']}\n"
        f"---\n{body[:3000]}"
    )

# ✅ Include path + symbol in the document side
# ❌ Feed raw embedding text without path — models confuse siblings

Hybrid lexical + vector recall before rerank still wins on exact identifiers — see OpenSearch vs Aurora pgvector.

Latency and cost budgets

# ✅ Canary: rerank must improve nDCG without blowing budget
pytest evals/test_rerank_lift.py --max-p95-ms=300 --min-ndcg-lift=0.08

# ❌ Rerank 500 candidates on every IDE keystroke

Emit rerank_lift (delta nDCG) and rerank_latency_ms alongside token budgets.

Closing checklist

✅ Dos
– ✅ Broad recall then cross-encoder top-N
– ✅ Put path/symbol in the document string
– ✅ Eval nDCG@k and faithfulness with/without rerank
– ✅ Cap candidate count for interactive UX
– ✅ Keep hybrid BM25 for exact symbols

❌ Don’ts
– ❌ Don’t ship vector-only top-k as “production RAG”
– ❌ Don’t rerank thousands of chunks synchronously in IDE paths
– ❌ Don’t ignore chunking quality — rerank cannot fix garbage chunks
– ❌ Don’t skip A/B on real developer questions
– ❌ Don’t forget tenant filters still wrap recall+rerank

Related reading

• Bedrock Knowledge Bases: Chunking Strategies That Fit Code RAG
• RAG Evaluation on AWS: Hit Rate, Faithfulness, and Cost Curves
• RAG on AWS: OpenSearch vs Aurora pgvector for Codebase Chat
• Bedrock Retrieval Filters: Tenant Isolation for Multi-SaaS Code RAG

Last updated on September 11, 2026