Day 19: Offline vs Online Evaluation

Offline suites (Day 9) stop known regressions. They do not see new ticket jargon, shifted runbooks, or prompt edits that only fail on live traffic. Day 19 pairs offline gates with online evaluation: canaries in CI plus sampled production traces scored for retrieval hit-rate, faithfulness, and refusal correctness.

⚡ TL;DR: Keep offline goldens as merge blockers. Sample production traces with privacy controls (Day 18). Score with the same metrics. Alert on drift before users file “the bot got dumb” tickets. Never train or grade on raw unrestricted PII.

Offline vs online responsibilities

# ✅ Sampled online scorer (pseudocode)
def on_response(trace):
    if hash(trace.request_id) % 100 >= SAMPLE_PCT:
        return
    redacted = redact(trace)
    offline_like = score(redacted)  # retrieval ids vs weak labels / judges
    metrics.write(offline_like)

Canary tickets in CI

Maintain 5–10 “canary” tickets that historically broke the bot (wrong service, ambiguous env, injection). Run them on every prompt/index change alongside the golden set. Canaries are regression magnets — treat them as precious.

Drift signals that matter

• Retrieval Recall proxy: fraction of answers whose top chunk shares a topic tag with the query classifier.
• Sudden refusal-rate collapse (bot became “helpful” = dangerous) or spike (bot became mute).
• Citation coverage drop on knowledge intents.
• Tool loop rate (Day 12) rising week over week.

✅ Alert: citation_coverage_knowledge < 0.95 for 2h
❌ Alert: “avg answer length changed” as your only quality signal

Privacy and sampling

Sample by request hash, not by “interesting failures only” (selection bias). Redact before any LLM judge. Drop traces from VIP/legal holds. Document the online eval data flow in the same packet as Day 18.

Field notes from production

When a model provider silently changes behavior behind the same ID, online charts move first. Offline goldens may still pass if they are too small. Keep a weekly human review of 20 random online samples — automation without spot checks decays.

Implementation sketch

# Compare online rolling 7d vs offline baseline
if online["faithfulness_7d"] < baseline["faithfulness"] - 0.05:
    page("rag-oncall", "online faithfulness drift")

Closing checklist

• [ ] Offline gate still required on merge
• [ ] Privacy-safe online sampling enabled
• [ ] Canary tickets in CI
• [ ] Drift alerts on citation/refusal/loop metrics
• [ ] Weekly human spot-check of samples
• [ ] Documented data flow for online evals

Labeling online samples

Use weak labels where possible: click engagement, whether user rephrased immediately, whether a human edited the bot’s PR comment. For high-value paths, route samples to a labeling queue with guidelines. Do not let LLM judges silently rewrite your quality definition week to week without a frozen rubric version. Compare online weak metrics to offline faithfulness monthly and investigate divergence.

Extended discussion

Return to the core angle for Day 19: Canary tickets in CI plus production traces sampled for drift. That sentence is the acceptance lens for every design review this week. If a proposed change does not make this angle easier to measure or enforce, it is a distraction.

Write down three metrics you will look at after shipping Day 19 ideas, schedule a 45-minute readout, and archive the notes next to the eval artifacts. Architecture without a readout becomes slideshow archaeology.

Pair this day with the adjacent lessons in the series navigation below. Forward links exist so you can keep momentum; backward links exist so you can repair foundations when a later lab fails for boring earlier reasons.

Practically, allocate half a day to implement the smallest vertical slice, half a day to wire measurement, and refuse to polish UI until both are done. This ordering is how bootcamp projects stay honest under time pressure.

Revisit assumptions whenever the model ID, embedding ID, or index alias changes — treat those as breaking changes for Day 19 behaviors, with the same seriousness as a database migration. Canary first, then promote.

Revisit assumptions whenever the model ID, embedding ID, or index alias changes — treat those as breaking changes for Day 19 behaviors, with the same seriousness as a database migration. Canary first, then promote.

Revisit assumptions whenever the model ID, embedding ID, or index alias changes — treat those as breaking changes for Day 19 behaviors, with the same seriousness as a database migration. Canary first, then promote.

Revisit assumptions whenever the model ID, embedding ID, or index alias changes — treat those as breaking changes for Day 19 behaviors, with the same seriousness as a database migration. Canary first, then promote.

Revisit assumptions whenever the model ID, embedding ID, or index alias changes — treat those as breaking changes for Day 19 behaviors, with the same seriousness as a database migration. Canary first, then promote.

Revisit assumptions whenever the model ID, embedding ID, or index alias changes — treat those as breaking changes for Day 19 behaviors, with the same seriousness as a database migration. Canary first, then promote.

Revisit assumptions whenever the model ID, embedding ID, or index alias changes — treat those as breaking changes for Day 19 behaviors, with the same seriousness as a database migration. Canary first, then promote.

Revisit assumptions whenever the model ID, embedding ID, or index alias changes — treat those as breaking changes for Day 19 behaviors, with the same seriousness as a database migration. Canary first, then promote.

Series navigation

← Day 18 · Day 20 →

Last updated September 11, 2026