Day 9: Eval Harness Day One

If you ship RAG or agents without an eval harness, you are unit-testing nothing and integration-testing vibes. Day 9 installs day-one evals: a golden question set, retrieval hit-rate, answer faithfulness, and a CI job that blocks merges when quality regresses.

⚡ TL;DR: Start with 30–100 labeled cases. Measure retrieval Recall@k and grounded faithfulness. Run on every prompt/index/model change in CI. Fail the build on statistically meaningful regressions — dashboards without gates are wallpaper.

Golden sets that reflect reality

Mine cases from real tickets and chat logs (sanitized). Each case needs:

• id, query, intent (knowledge / navigate / act)
• relevant_chunk_ids (for retrieval)
• rubric or reference answer notes (for generation)
• must_refuse flag for out-of-corpus / unsafe asks

# ✅ Minimal case schema
case = {
  "id": "rds-failover-01",
  "query": "How long does Multi-AZ RDS failover usually take for checkout?",
  "relevant_chunk_ids": ["runbook:rds-failover:v3"],
  "must_cite": True,
  "must_refuse": False,
}

Refresh monthly. Stale goldens certify a dead corpus.

Metrics that change decisions

# ✅ CI gate sketch
def main():
    base = load_json("eval/baseline.json")
    cur = run_suite(cases)
    if cur["recall@10"] < base["recall@10"] - 0.03:
        raise SystemExit("recall regression")
    if cur["faithfulness"] < base["faithfulness"] - 0.03:
        raise SystemExit("faithfulness regression")
    print("eval ok", cur)

❌ Only measuring “average thumbs-up in Slack” after deploy.

Offline judges without self-delusion

LLM-as-judge is useful for faithfulness if you constrain it: provide evidence, ask “supported / unsupported / contradictory,” and spot-check with humans. Do not let the same model grade its own unconstrained creativity without evidence packs.

For coding agents, prefer deterministic oracles: tests pass, cdk synth succeeds, policy linter clean — not “looks good.”

Wire it into CI and release

• On PR: cheap subset (smoke 20 cases).
• Nightly: full suite + canaries.
• On model/prompt/index change: mandatory full suite.
• Store results as artifacts; update baseline only via intentional PR.

Pair with Day 19 for online evaluation; offline is the merge gate, online catches drift.

Closing checklist

• [ ] ≥30 labeled golden cases checked into the repo
• [ ] Retrieval and generation metrics defined with thresholds
• [ ] CI job fails on regression vs committed baseline
• [ ] Sanitized real queries, not only synthetic toys
• [ ] Monthly golden-set review with on-call
• [ ] Artifacts retained for incident retrospectives

Worked example: baseline PR discipline

Commit eval/baseline.json. Only update it in a PR titled eval: accept new baseline with commentary (model upgrade, corpus expansion). Casual baseline bumps to silence CI are process failures equal to deleting unit tests.

Failure modes to watch

• Synthetic-only goldens that never match production jargon.
• LLM judge without evidence grading fluency as correctness.
• Nightly-only evals with no PR smoke gate.
• Metrics without owners.

Field notes from production

Owners: name a human for the golden set. Without an owner, baselines rot. Schedule a 30-minute monthly review after major incidents — add the queries that hurt you. Publish eval trends next to error budgets so leadership sees quality as reliability.

Implementation sketch

# Implementation sketch: CI job
# .github/workflows/eval.yml
# run: python -m eval.run --baseline eval/baseline.json --fail-under

Operator addendum

Keep a ‘known flaky’ quarantine for eval cases under investigation — but cap it. Quarantines without expiry become places where quality goes to die, just like flaky unit tests.

Building goldens without bias

Sample tickets proportional to production intents; do not only pick questions your current bot already answers. Include must-refuse and must-escalate cases. Have a second engineer verify labels. Store rationales (“chunk C3 because it states the 60s RTO”) so future editors do not relabel casually. When the corpus changes, mark cases needs_revisit instead of silently failing forever or deleting history. Publish a monthly quality note: what improved, what regressed, what you quarantined.

Extended discussion

Return to the core angle for Day 9: Golden questions, faithfulness, hit-rate, and a CI job that fails the build. That sentence is the acceptance lens for every design review this week. If a proposed change does not make this angle easier to measure or enforce, it is a distraction.

Write down three metrics you will look at after shipping Day 9 ideas, schedule a 45-minute readout, and archive the notes next to the eval artifacts. Architecture without a readout becomes slideshow archaeology.

Pair this day with the adjacent lessons in the series navigation below. Forward links exist so you can keep momentum; backward links exist so you can repair foundations when a later lab fails for boring earlier reasons.

Practically, allocate half a day to implement the smallest vertical slice, half a day to wire measurement, and refuse to polish UI until both are done. This ordering is how bootcamp projects stay honest under time pressure.

Revisit assumptions whenever the model ID, embedding ID, or index alias changes — treat those as breaking changes for Day 9 behaviors, with the same seriousness as a database migration. Canary first, then promote.

Series navigation

← Day 8 · Day 10 →

Last updated September 11, 2026