Day 86: Search and Ranking Copilots

Search copilots that “feel better” on three queries will regress relevance in silence. Treat ranking changes like model launches: offline deltas, online interleaving, and kill switches.

⚡ TL;DR: Build a labeled query set. Measure NDCG/Recall@k offline. Interleave online. Ship ranker/prompt changes behind flags. Ban anecdote-only launches.

Eval loop

# ✅ Offline delta before online
def ndcg_delta(control_ranker, candidate_ranker, cases) -> float:
    c = mean(ndcg(control_ranker(c.query), c.relevant) for c in cases)
    n = mean(ndcg(candidate_ranker(c.query), c.relevant) for c in cases)
    return n - c

def ship_ok(delta: float, p_value: float) -> bool:
    return delta >= 0.01 and p_value < 0.05

❌ Hot-swapping a reranker prompt in prod because a PM disliked one result.

Failure modes

Training the ranker on click logs that encode position bias. Use counterfactual or interleave methods. Product managers cherry-picking three queries to force a ship — require the eval gate in writing.

Closing checklist

• [ ] Golden query set versioned in git
• [ ] Offline + online gates documented
• [ ] Feature flag for ranker
• [ ] Latency regression tests
• [ ] Rollback runbook

Series navigation

Day 85: Time-Series and Anomaly Copilots · Day 87: Workflow Mining From Logs

Last updated September 11, 2026