Day 41: When Fine-Tunes Beat Prompting

Prompting is cheap until the prompt is a novella and still mis-formats your internal SDK. Fine-tuning wins when behavior must be stable, terse, and domain-hard — and when you have labeled data, not vibes. Day 41 is the decision gate before you burn GPU budget.

⚡ TL;DR: Fine-tune for format/dialect lock-in, private vocabulary, or smaller/faster students. Stay on prompts+RAG when tasks change weekly or curated pairs < ~1k. Compare Bedrock customization vs SageMaker JumpStart by ops ownership. Require a baseline metric first.

Decision tree

1. ≥1k curated examples matching production inputs? If no → prompt/RAG.
2. Failure mode “wrong format / wrong API dialect” more than “missing facts”? Facts → RAG; dialect → fine-tune.
3. Can you run blind A/B on real tickets (Day 49)? If no → not ready.

decision = {
  "problem": "sdk_autocomplete_wrong_methods",
  "n_train": 4200,
  "n_eval": 600,
  "baseline_prompt_score": 0.61,
  "go_if_expected_lift": 0.08,
}

❌ "Leadership wants our own model" with 80 Slack screenshots

Bedrock vs JumpStart

Mine accepted PR fixes, corrected agent patches, and cited Q&A. Exclude secrets (Day 48). Prefer diffs + final code over hedge-filled chat logs. Include labeling hours in the cost — a 2-point win that costs a team-month may lose to better retrieval (Days 26–30).

Production checklist

• [ ] Decision record with baseline metrics
• [ ] Data inventory + license review started
• [ ] Eval pack frozen before training
• [ ] Rollback to prompt-only defined
• [ ] Owner for drift monitoring
• [ ] Cost model includes labeling + serving

Series navigation

← Day 40 · Day 42 →

Last updated September 11, 2026