Day 42: Preference Data Without Vanity Accept Rates

Thumbs-up rates climb when the model is agreeable and wrong. Preference data for coding assistants must prefer correct diffs and cited answers, not nicer tone. Day 42 defines pairwise rows grounded in CI and merges.

⚡ TL;DR: Collect (prompt, chosen, rejected) from real tickets where chosen passed CI/human merge. Demote UI thumbs to secondary. Audit for sycophancy. Feed DPO-style trainers only after validators pass.

Preference row shape

{
  "id": "pref-9182",
  "prompt": "Fix flaky auth middleware test",
  "chosen": {"type": "diff", "text": "...", "ci": "passed", "merged": true},
  "rejected": {"type": "diff", "text": "...", "ci": "failed"},
  "policy_tags": ["correctness", "minimal_diff"]
}

Good sources: agent patch vs human final patch; cite-gated answer vs uncitable waffle; two candidates judged by offline tests. Rater guidelines: prefer smaller correct diffs, existing style, no drive-by refactors; reject invented files and deleted tests.

Production checklist

• [ ] Preference schema versioned
• [ ] CI outcome attached when available
• [ ] Thumbs UI demoted
• [ ] Inter-rater audits monthly
• [ ] Sycophancy probes in eval pack
• [ ] Holdout never shown model identity (Day 49)

Series navigation

← Day 41 · Day 43 →

Last updated September 11, 2026