Not every problem needs a full SFT run. Continued pretraining teaches domain tokens; instruct tuning teaches tasks; adapters give cheap specialization. Day 44 picks the cheapest lever that moves the metric.
⚡ TL;DR: CPT for jargon-heavy corpora; SFT for task format; LoRA for multi-tenant specialists. Measure lift per GPU-hour. Always re-check forgetting (Day 45).
| Lever | Teaches | Cost | Risk |
|---|---|---|---|
| CPT | domain distribution | high | forgetting |
| Instruct/SFT | task specs | medium | overfit style |
| Adapters/LoRA | specialist behavior | low | routing complexity |
| Prompt+RAG | facts | lowest | context limits |
experiments = [
{"lever": "lora", "hours": 4, "metric": "sdk_exact_api", "hyp_lift": 0.05},
{"lever": "sft", "hours": 20, "metric": "sdk_exact_api", "hyp_lift": 0.08},
{"lever": "cpt", "hours": 80, "metric": "sdk_exact_api", "hyp_lift": 0.03},
]
❌ Jump to full CPT because a blog said "domain adaptation"
Sequence: freeze eval → try LoRA → consider CPT only if tokenization/jargon is the bottleneck → always run forgetting checks.
Production checklist
- [ ] Lever choice + hypothesized lift recorded
- [ ] GPU/Bedrock hours budgeted
- [ ] Same eval for all levers
- [ ] Adapter registry versioned
- [ ] Kill criteria if lift < cost threshold
- [ ] Multi-tenant adapter isolation reviewed
Series navigation
Last updated September 11, 2026
Discover more from CheatCoders
Subscribe to get the latest posts sent to your email.
