Day 45: Catastrophic Forgetting Checks

Domain fine-tunes go feral: they ace your SDK suite and forget try/finally. Without a hold-out general pack, you ship a specialist idiot. Day 45 installs the gate.

⚡ TL;DR: Maintain a frozen general coding eval. Block deploy if general score drops beyond budget even when domain score rises. Plot base vs candidate on both axes.

def gate(scores: dict, budgets: dict) -> bool:
    if scores["domain"] < budgets["domain_min"]:
        return False
    if scores["general"] < budgets["general_floor"]:
        return False
    if scores["general"] < budgets["general_baseline"] - budgets["max_drop"]:
        return False
    return True

General pack contents: language fundamentals, common libraries you still use, security basics, and “don’t delete tests.” Keep it stable — churning the pack hides regressions. Archive base checkpoint scores forever.

Production checklist

• [ ] Frozen general pack versioned
• [ ] Automated gate in training CI
• [ ] Max drop budget agreed
• [ ] Failure examples ticketed
• [ ] Base scores archived
• [ ] Pareto plots in the release notes

Series navigation

← Day 44 · Day 46 →

Last updated September 11, 2026