Day 89: On-Device and Edge Completions

Cloud models are default — until privacy, offline laptops, or 30ms autocomplete constraints say otherwise. On-device and edge completions should be the default path for local IDE ghost text, with cloud as escalation for hard tasks.

⚡ TL;DR: Route by task class. Tiny local model for complete-current-line; cloud for multi-file refactors. Version local models; measure quality floors; never send secrets off-box when local suffices.

Routing policy

def route(task: dict) -> str:
    if task["kind"] == "line_complete" and task["secret_risk"] == "low":
        return "local_small"
    if task.get("needs_repo_rag"):
        return "cloud_rag"
    if task["kind"] == "multi_file_refactor":
        return "cloud_large"
    return "local_small"

Failure modes

Stale on-device models after a breaking tokenizer change. Pin model+tokenizer pairs and fail closed if checksum mismatches. Users disabling local path “because cloud is smarter” for secrets-laden files — enforce local for high secret_risk.

Closing checklist

• [ ] Task-based router, not one model forever
• [ ] Local model update channel + checksums
• [ ] Quality floor evals for local path
• [ ] Secret-aware: prefer local when diff has keys scrubbed poorly
• [ ] UX shows which path answered

Series navigation

Day 88: Simulation and Digital Twins for Tools · Day 90: Project: Incident Timeline Summarizer

Last updated September 11, 2026