26 views
This guide covers LLM model routing with production-focused examples and trade-offs.
Not every agent turn deserves a frontier model. Route boilerplate and mechanical patches to cheap draft models; reserve expensive verifiers for architecture review, security-sensitive diffs, and flaky-test RCA. You cut spend without lowering the bar on judgments that protect production.⚡ TL;DR: Classify tasks (draft vs verify); run Haiku/Sonnet-class (or local) drafts; escalate to frontier only on risk signals (IAM, auth, migrations, public API, flaky flakes). Cache shared system prompts. Measure $ / merged PR and revert rate. Companions: prompt caching, code review bots, Cursor rules.
Task taxonomy — LLM model routing
typescript
✅ Explicit classifier you can unit test.export type RouteClass = "draft" | "verify" | "forbid_auto";
export function classify(job: {
files: string[];
labels: string[];
locDelta: number;
}): RouteClass {
const sensitive = job.files.some(f =>
/(^|\/)(iam|auth|migration|infra)\//i.test(f) ||
/\.github\/workflows\//.test(f)
);
if (sensitive || job.labels.includes("security")) return "verify";
if (job.labels.includes("public-api")) return "verify";
if (job.locDelta > 400) return "verify";
if (job.files.some(f => f.includes("break-glass"))) return "forbid_auto";
return "draft";
}
❌ “Let the agent decide if it needs a bigger model.”
Two-pass pipeline
python
Verifier system prompt should focus on: authz bugs, IAM wildcards, missing tests, API breakage — not rewriting style the draft already handled.
DRAFT_MODEL = "anthropic.claude-haiku-4-5-20251001-v1:0"
VERIFY_MODEL = "anthropic.claude-sonnet-4-20250514-v1:0"
def handle(job):
route = classify(job)
if route == "forbid_auto":
return needs_human(job)
draft = converse(DRAFT_MODEL, build_draft_prompt(job), tool_choice="auto")
patch = execute_tools(draft)
if route == "draft" and not risk_signals(patch):
return patch # cheap path
# Expensive verifier: read-only review, may reject
verdict = converse(VERIFY_MODEL, build_verify_prompt(job, patch), tool_choice={"auto": {}})
if verdict_rejects(verdict):
return revise_or_human(job, verdict)
return patch
Risk signals that force escalation
typescript
Wire the same patterns into AI Code Review Bots so human PRs get equivalent scrutiny.
const RISK = [
/Resource\s*:\s*"\*"/,
/Action\s*:\s*"\*"/,
/ignore.*(auth|tenant)/i,
/password|secret|AKIA/,
/DROP TABLE|prisma\s+migrate\s+reset/i,
];
export function riskSignals(diff: string): boolean {
return RISK.some(r => r.test(diff));
}
Economics (illustrative)
text
Track: tokens by model id, escalations / day, post-merge incidents tagged # Planning examples — not a vendor quote
# 80% of turns on draft model @ ~0.2–0.4× frontier price
# 20% verify @ full price
# Blended often lands ~0.35–0.55× all-frontier with similar revert rates
# Prompt caching on shared rules/schemas amplifies savings further
agent-authored.
Closing checklist
✅ Dos – ✅ Classify draft vs verify with tested rules – ✅ Escalate on IAM/auth/migrations/public API/large diffs – ✅ Keep verifiers read-heavy (review), drafts write-heavy – ✅ Log model id per turn for cost attribution – ✅ Combine with prompt caching on shared system text ❌ Don’ts – ❌ Don’t run security diffs only on the cheap model – ❌ Don’t hide escalations from developers – ❌ Don’t use verifier temperature > 0 for CI gates – ❌ Don’t skip human review onforbid_auto paths
– ❌ Don’t optimize cost without watching revert/incident rates
Related reading
- Bedrock Prompt Caching and Batch Inference: Cut Latency and Cost
- AI Code Review Bots: IAM, Secrets, and Least-Privilege Pipelines
- Cursor Rules for TypeScript Monorepos: Make AI Edits Stick
- Amazon Bedrock Agents: Tool Use, Memory, and Production Guardrails
Related: Day 73: Draft-Then-Verify Routing · Day 22: Rerankers After Cheap Recall · Day 1: Tokens, Context Windows, and Why Models Forget Mid-Task
Related: Multi-Agent Systems: Orchestrating Claude, GPT-4o, and Specialized Models Together · Code RAG Rerankers: Cross-Encoders After Cheap Vector Recall · Incident ChatOps AI: Draft Change Tickets Without Auto-Remediating
Watch: 100 Days of AI on YouTube
Short videos from CheatCoders, one AI engineering topic per day.
Most viewed
- Python Decorators Explained: From Simple Wrappers to Production Patterns
- Claude Code vs Cursor vs GitHub Copilot: Choose by Workflow, and Set Up One Repo That Works With All Three
- AI Agent Frameworks in 2025: LangGraph vs CrewAI vs AutoGen vs Raw API
- Python String Methods: Every str Method With Real Production Examples
- Css Animation Performance: Performance-Optimized Motion for Modern UIs
Newly added
- AWS Systems Manager Parameter Store: Hierarchical Runtime Config for Coding Agents
- Amazon Cognito Identity Pools: Federate Coding-Agent Tool Callers Without Long-Lived Keys
- Amazon Athena for Coding Agents: Query Tool Audit Logs and Eval Traces on S3 Without a Warehouse
- AWS Batch for Coding Agents: Parallel Overnight PR Reviews and Long-Running Jobs Without Lambda Timeouts
- Amazon Bedrock Prompt Management: Version and A/B Test Coding-Agent System Prompts
Deep-dive PDF
Get the expanded guide for this post — extra diagrams-style checklists, failure modes, and a production walkthrough. Free when you subscribe to CheatCoders.
Already subscribed? or open the subscribe page.
Discover more from CheatCoders
Subscribe to get the latest posts sent to your email.
