Day 22: Rerankers After Cheap Recall

Retrieve broadly, then rerank the shortlist with a cross-encoder or managed rerank API. Day 22 places rerankers in the pipeline correctly: after cheap recall (hybrid Day 21), before generation — and teaches when they are not worth the p95 latency tax.

⚡ TL;DR: Candidate generate with hybrid top-50–100. Rerank to top-5–10. Measure delta Recall@10 / nDCG vs p95. Disable rerank on ultra-low-latency paths if gains are < threshold. Cache rerank results for identical queries when safe.

Pipeline placement

query → hybrid recall (k=80) → rerank (top 8) → generate

# ✅ Conditional rerank
cands = hybrid(query, tenant, k=80)
if latency_class == "interactive_tight" and not force_quality:
    return cands[:8]
return bedrock_rerank(query, cands, top_n=8)

❌ Cross-encoding the entire corpus — that is not a reranker, that is a denial of service.

When rerankers win

• Ambiguous short queries with many near-neighbors.
• Mixed keyword/semantic needs where fusion still leaves noise.
• High-stakes answers where an extra 100–300ms is acceptable.

When they lose:

• Already precise symbol hits at rank 1.
• Hard interactive SLOs (<300ms total retrieve).
• Tiny corpora where exact ranking barely moves.

Bedrock and self-hosted options

Managed rerank APIs reduce ops. Self-hosted cross-encoders give control and cost predictability at scale. Either way, version the reranker ID like embeddings (Day 3) and include it in traces (Day 5).

Evaluate with and without rerank on the same candidate sets — isolate the gain.

Field notes from production

Rerank cache keys must include model ID and candidate ID set hash. Stale caches after corpus updates cause eerie “fixed” rankings. Invalidate on re-ingest aliases.

Implementation sketch

def retrieve(query, tenant, slo_ms):
    cands = hybrid(query, tenant, k=80)
    if slo_ms < 400:
        return cands[:8]
    return rerank(query, cands, top_n=8)

Closing checklist

• [ ] Rerank after hybrid recall, never alone
• [ ] A/B nDCG vs p95 documented
• [ ] Escape hatch for tight latency classes
• [ ] Reranker version in traces
• [ ] Cache invalidation on corpus change
• [ ] Skip rerank when symbol exact-hit confidence high

Cost of rerank at scale

Multiply cross-encoder latency by QPS and candidate count. At 50 QPS × 80 candidates, you are in serious compute territory if self-hosting. Managed APIs trade ops for unit cost — model the monthly bill at p95 traffic, not averages. Consider reranking only when top-1 hybrid score margin is thin (ambiguous). That adaptive policy often keeps quality gains while protecting SLOs.

Extended discussion

Return to the core angle for Day 22: Cross-encoders, Bedrock rerank, and when they are not worth the p95. That sentence is the acceptance lens for every design review this week. If a proposed change does not make this angle easier to measure or enforce, it is a distraction.

Write down three metrics you will look at after shipping Day 22 ideas, schedule a 45-minute readout, and archive the notes next to the eval artifacts. Architecture without a readout becomes slideshow archaeology.

Pair this day with the adjacent lessons in the series navigation below. Forward links exist so you can keep momentum; backward links exist so you can repair foundations when a later lab fails for boring earlier reasons.

Practically, allocate half a day to implement the smallest vertical slice, half a day to wire measurement, and refuse to polish UI until both are done. This ordering is how bootcamp projects stay honest under time pressure.

Revisit assumptions whenever the model ID, embedding ID, or index alias changes — treat those as breaking changes for Day 22 behaviors, with the same seriousness as a database migration. Canary first, then promote.

Revisit assumptions whenever the model ID, embedding ID, or index alias changes — treat those as breaking changes for Day 22 behaviors, with the same seriousness as a database migration. Canary first, then promote.

Revisit assumptions whenever the model ID, embedding ID, or index alias changes — treat those as breaking changes for Day 22 behaviors, with the same seriousness as a database migration. Canary first, then promote.

Revisit assumptions whenever the model ID, embedding ID, or index alias changes — treat those as breaking changes for Day 22 behaviors, with the same seriousness as a database migration. Canary first, then promote.

Revisit assumptions whenever the model ID, embedding ID, or index alias changes — treat those as breaking changes for Day 22 behaviors, with the same seriousness as a database migration. Canary first, then promote.

Revisit assumptions whenever the model ID, embedding ID, or index alias changes — treat those as breaking changes for Day 22 behaviors, with the same seriousness as a database migration. Canary first, then promote.

Revisit assumptions whenever the model ID, embedding ID, or index alias changes — treat those as breaking changes for Day 22 behaviors, with the same seriousness as a database migration. Canary first, then promote.

Revisit assumptions whenever the model ID, embedding ID, or index alias changes — treat those as breaking changes for Day 22 behaviors, with the same seriousness as a database migration. Canary first, then promote.

Series navigation

← Day 21 · Day 23 →

Last updated September 11, 2026