Most teams stare at output token prices while input tokens quietly dominate agent and RAG bills: every tool schema, every retrieved chunk, every replayed transcript. Day 15 builds cost curves you can act on — prompt caching, draft/small models, and per-PR or per-team budgets visible to engineers.
⚡ TL;DR: Attribute $ to input vs output by feature. Stabilize prefixes for cache hits. Route easy tasks to smaller/cheaper models. Publish weekly token budgets per team with hard stops or soft gates in CI.
Draw the curve
# ✅ Unit economics per successful task
cost = in_tokens * in_price + out_tokens * out_price
print({"task": task_id, "in": in_tokens, "out": out_tokens, "usd": cost})
Plot cost versus retrieval k, versus history length, versus tool-loop depth. You will usually find a knee where another chunk or step adds cost without lifting eval scores (Day 9).
Levers that move the bill
| Lever | Saves | Risk |
|---|---|---|
| Prompt caching / stable prefixes | Input on repeated system+tools | Cache misses from volatile prefixes |
| Smaller draft model | Exploration tokens | Quality — gate with evals |
Lower k / better rerank |
RAG input | Recall — measure |
| Summarize history | Session input | Lost detail |
| Cap agent steps | Loops | Incomplete tasks |
✅ Interactive: Haiku/Sonnet-class for classify + retrieve; larger model only for final patch
❌ Opus-class for every “label this intent” call
Budgets engineers can see
Put token burn on the PR check or team dashboard: “this agent run used $0.42 / budget $1.00.” Soft warn at 70%, hard fail optional for non-prod sandboxes. Budgets without visibility become finance lectures nobody reads.
On Bedrock, track by application inference profile / tags so platform can chargeback.
Closing checklist
- [ ] Cost split input vs output per feature
- [ ] Cacheable stable prefixes
- [ ] Tiered models by task difficulty
- [ ] Team or PR token budgets visible in CI/UI
- [ ] Knee charts vs
kand max steps - [ ] Kill switches when daily spend exceeds cap
Worked example: RAG k sweep vs $
Run goldens at k=4,8,16. If Recall@10 plateaus at 8 but cost doubles at 16, lock k=8. Publish the chart in the README so a well-meaning engineer does not “just raise k” next quarter.
Failure modes to watch
- FinOps panic without product attribution.
- Caches disabled because someone put
datetime.utcnow()in the system prompt. - One model tier for everything.
- Budgets only in spreadsheets updated monthly.
Field notes from production
Chargeback by team and feature tags on every call. Without attribution, platform eats the bill and cannot prioritize caching work. Publish a weekly top-10 most expensive prompt templates — shame is a surprisingly effective optimizer.
Implementation sketch
# Implementation sketch: budget header for PR bots
def within_budget(pr, usd):
spent = metrics.sum(pr=pr, days=1)
return spent + usd <= budgets.team_daily
Operator addendum
Make the expensive prompt templates searchable. A single forgotten debug flag that injects full file trees can dominate monthly spend more than model price changes.
Model routing policy
Document a routing table: intent → model tier → max input tokens → max steps. Review it when prices change. Example: classify/retrieve planning on a cheap model; final patch on a stronger model; batch eval on provisioned throughput overnight. A/B the router with evals before global rollout. Attach estimated USD to each row so product managers can argue with numbers, not folklore.
Extended discussion
Return to the core angle for Day 15: Caching, draft models, and per-PR budgets that engineers can see. That sentence is the acceptance lens for every design review this week. If a proposed change does not make this angle easier to measure or enforce, it is a distraction.
Write down three metrics you will look at after shipping Day 15 ideas, schedule a 45-minute readout, and archive the notes next to the eval artifacts. Architecture without a readout becomes slideshow archaeology.
Pair this day with the adjacent lessons in the series navigation below. Forward links exist so you can keep momentum; backward links exist so you can repair foundations when a later lab fails for boring earlier reasons.
Practically, allocate half a day to implement the smallest vertical slice, half a day to wire measurement, and refuse to polish UI until both are done. This ordering is how bootcamp projects stay honest under time pressure.
Revisit assumptions whenever the model ID, embedding ID, or index alias changes — treat those as breaking changes for Day 15 behaviors, with the same seriousness as a database migration. Canary first, then promote.
Revisit assumptions whenever the model ID, embedding ID, or index alias changes — treat those as breaking changes for Day 15 behaviors, with the same seriousness as a database migration. Canary first, then promote.
Revisit assumptions whenever the model ID, embedding ID, or index alias changes — treat those as breaking changes for Day 15 behaviors, with the same seriousness as a database migration. Canary first, then promote.
Revisit assumptions whenever the model ID, embedding ID, or index alias changes — treat those as breaking changes for Day 15 behaviors, with the same seriousness as a database migration. Canary first, then promote.
Revisit assumptions whenever the model ID, embedding ID, or index alias changes — treat those as breaking changes for Day 15 behaviors, with the same seriousness as a database migration. Canary first, then promote.
Series navigation
Last updated September 11, 2026
Discover more from CheatCoders
Subscribe to get the latest posts sent to your email.
