Day 49: Eval for Fine-Tunes: Blind A/B on Real Tickets

Offline loss is necessary; insufficient. Blind A/B on real developer tickets tells you if cycle time or defect rate moved. If not, keep the teacher. Day 49 pre-registers metrics and blinding.

⚡ TL;DR: Sample real tickets; two arms behind the router; raters blinded; primary metric time-to-merge or escaped defects. Either offline general-pack or online A/B can veto.

experiment = {
  "n_tickets": 120,
  "arms": ["teacher", "student_v3"],
  "blinding": True,
  "primary": "time_to_merge_hours",
  "secondary": ["ci_fail_rate", "revert_rate", "rater_preference"],
  "min_detectable_effect_hours": 0.5,
}

❌ Ship because BLEU vs teacher improved

Router assigns arm server-side; UI chrome identical. Break blind only after metrics lock. Ship only if primary beats MDE, general pack within budget (Day 45), and license scan clean (Day 48).

Production checklist

• [ ] Pre-registered metrics
• [ ] Sample size / MDE documented
• [ ] Blinding procedure
• [ ] Stop/rollback criteria
• [ ] Results archived with model hashes
• [ ] Decision recorded even when “no ship”

Series navigation

← Day 48 · Day 50 →

Last updated September 11, 2026