Day 21: Hybrid Search: BM25 + Vectors + Symbols

Dense vectors miss exact identifiers; BM25 misses paraphrases; symbol indexes miss prose. Day 21 builds hybrid search for code and ops docs: BM25 + vectors + symbol/lookup channels fused into one ranking that beats any single channel on real coding queries.

⚡ TL;DR: Run BM25, ANN, and symbol match in parallel. Fuse with RRF or learned weights. Keep channels’ scores logged. Evaluate on queries that need exact error codes and conceptual matches. Never drop the keyword channel for code RAG.

Three complementary signals

# ✅ Reciprocal Rank Fusion (simple, strong baseline)
def rrf(rank_lists, k=60):
    scores = {}
    for ranks in rank_lists:
        for r, doc_id in enumerate(ranks, start=1):
            scores[doc_id] = scores.get(doc_id, 0) + 1.0 / (k + r)
    return sorted(scores, key=scores.get, reverse=True)

❌ Averaging raw BM25 and cosine as if they share a scale.

Symbol channel for code

Maintain an extractor: functions, classes, routes, IAM action strings, Terraform addresses. Exact hit should boost hard — often as a hard filter or top-boost before fusion.

Query: "ExpiredToken when calling AssumeRole"
BM25: hits boto error docs
Vector: hits SSO refresh runbook
Symbol: hits helper refresh_sso_token()
Fusion: all three → correct fix path

Tuning without folklore

Sweep fusion k and channel enables on the Day 3/9 labeled set. Some query classes (pure conceptual) want vectors heavier; pure identifier queries want BM25/symbol. A cheap query classifier can pick fusion weights.

OpenSearch supports hybrid queries; pgvector + Postgres FTS can fuse in app code. Pick ops you can staff.

Field notes from production

Log which channel contributed the winning chunk. If vector always wins and BM25 never does, your BM25 analyzer is broken (stemming identifiers into mush) — fix analysis, do not delete the channel.

Implementation sketch

def hybrid(query, tenant):
    q = classify(query)  # identifier_heavy | conceptual | mixed
    bm = bm25(query, tenant, k=50)
    ann = ann_search(embed(query), tenant, k=50)
    sym = symbol_lookup(query, tenant, k=20)
    weights = WEIGHTS[q]
    return fuse(bm, ann, sym, weights)[:20]

Closing checklist

• [ ] BM25 + vector + symbol channels live
• [ ] RRF or tuned fusion — not naive score average
• [ ] Analyzer that preserves code tokens
• [ ] Eval slices for identifier vs conceptual queries
• [ ] Per-result channel attribution in logs
• [ ] Tenant filters on every channel (Day 23)

Analyzer configuration for code

Disable aggressive stemming on identifier fields; use keyword subfields for exact matches. Keep a text field for prose comments. Explain this in the index template README. Many “hybrid underperforms” reports are actually analyzer self-owns. Add regression tests that search for DoNotStemThisSymbol and expect an exact hit.

Extended discussion

Return to the core angle for Day 21: Fusion ranks that beat either signal alone on code RAG. That sentence is the acceptance lens for every design review this week. If a proposed change does not make this angle easier to measure or enforce, it is a distraction.

Write down three metrics you will look at after shipping Day 21 ideas, schedule a 45-minute readout, and archive the notes next to the eval artifacts. Architecture without a readout becomes slideshow archaeology.

Pair this day with the adjacent lessons in the series navigation below. Forward links exist so you can keep momentum; backward links exist so you can repair foundations when a later lab fails for boring earlier reasons.

Practically, allocate half a day to implement the smallest vertical slice, half a day to wire measurement, and refuse to polish UI until both are done. This ordering is how bootcamp projects stay honest under time pressure.

Revisit assumptions whenever the model ID, embedding ID, or index alias changes — treat those as breaking changes for Day 21 behaviors, with the same seriousness as a database migration. Canary first, then promote.

Revisit assumptions whenever the model ID, embedding ID, or index alias changes — treat those as breaking changes for Day 21 behaviors, with the same seriousness as a database migration. Canary first, then promote.

Revisit assumptions whenever the model ID, embedding ID, or index alias changes — treat those as breaking changes for Day 21 behaviors, with the same seriousness as a database migration. Canary first, then promote.

Revisit assumptions whenever the model ID, embedding ID, or index alias changes — treat those as breaking changes for Day 21 behaviors, with the same seriousness as a database migration. Canary first, then promote.

Revisit assumptions whenever the model ID, embedding ID, or index alias changes — treat those as breaking changes for Day 21 behaviors, with the same seriousness as a database migration. Canary first, then promote.

Revisit assumptions whenever the model ID, embedding ID, or index alias changes — treat those as breaking changes for Day 21 behaviors, with the same seriousness as a database migration. Canary first, then promote.

Revisit assumptions whenever the model ID, embedding ID, or index alias changes — treat those as breaking changes for Day 21 behaviors, with the same seriousness as a database migration. Canary first, then promote.

Revisit assumptions whenever the model ID, embedding ID, or index alias changes — treat those as breaking changes for Day 21 behaviors, with the same seriousness as a database migration. Canary first, then promote.

Series navigation

← Day 20 · Day 22 →

Last updated September 11, 2026