15 views
This guide focuses on RAG answer caching for production systems, with practical trade-offs for reliability, security, and cost.
Caching RAG answers is a sharp knife. Done right, identical internal questions stop burning tokens. Done wrong, Tenant A sees Tenant B’s answer, or yesterday’s wrong runbook ships forever after a git push.
⚡ TL;DR: Cache only normalized question + tenant + corpus version + model/prompt version. Short TTLs. Never cache across tenants. Invalidate on index refresh. Prefer caching retrieval sets over final prose when answers must cite live code.
RAG answer caching: production guidance
python
❌ Keying only on the question string. That is how you leak between customers and environments.
# ✅ Tenant-scoped, versioned cache key
import hashlib, json
def rag_cache_key(*, tenant_id: str, question: str, corpus_ver: str,
prompt_ver: str, model_id: str) -> str:
norm = " ".join(question.lower().split())
payload = {
"t": tenant_id,
"q": norm,
"c": corpus_ver, # git SHA or KB ingestion id
"p": prompt_ver,
"m": model_id,
}
raw = json.dumps(payload, sort_keys=True).encode()
return "rag:" + hashlib.sha256(raw).hexdigest()
What to cache
| Layer | Cache? | Notes |
|---|---|---|
| Embedding of query | Yes | Per embedding model version |
| Retrieval hit list | Often | Invalidate with corpus_ver |
| Final natural-language answer | Sometimes | Only for FAQ-like, cited answers |
| Tool side effects | Never | Not a cache problem |
Invalidation that actually runs
Hook your indexer (Day 27) to bumpcorpus_ver and delete keys by prefix or generation tag:
python
# ✅ Generation-based invalidation
def bump_corpus(tenant_id: str) -> str:
ver = new_version() # monotonic or git SHA
redis.set(f"corpus_ver:{tenant_id}", ver)
# optional: redis.scan_iter(f"rag:{tenant_id}:*") and delete
return ver
Failure modes
Cross-env cache hits (staging key reused in prod) are as bad as cross-tenant. Includeenv in the key. Clock-skewed TTLs across gateway replicas can resurrect deleted answers — prefer generation counters over wall-clock alone when corpus_ver is unchanged but policy prompts changed.
Closing checklist
- [ ] Tenant id mandatory in every cache key
- [ ] corpus_ver + prompt_ver + model_id in key
- [ ] TTL ≤ your freshness SLO (often hours, not weeks)
- [ ] Integration test: same question, two tenants → two entries
- [ ] Never serve cached answer without citations still present in index
Series navigation
Day 76: p99 of Agents: Queueing, Not Just Model Latency · Day 78: Load Shedding When the Model Is Sick Last updated September 11, 2026 Related: Cost-Aware RAG Caches: Reuse Answers for Identical Code Questions; Citation-Required RAG Answers: Force Models to Quote File Paths; RAG Evaluation on Bedrock: Offline Suites Before You Trust Code Answers.Watch: 100 Days of AI on YouTube
Short videos from CheatCoders, one AI engineering topic per day.
Most viewed
- Python Decorators Explained: From Simple Wrappers to Production Patterns
- Claude Code vs Cursor vs GitHub Copilot: Choose by Workflow, and Set Up One Repo That Works With All Three
- AI Agent Frameworks in 2025: LangGraph vs CrewAI vs AutoGen vs Raw API
- Python String Methods: Every str Method With Real Production Examples
- Css Animation Performance: Performance-Optimized Motion for Modern UIs
Newly added
- AWS Systems Manager Parameter Store: Hierarchical Runtime Config for Coding Agents
- Amazon Cognito Identity Pools: Federate Coding-Agent Tool Callers Without Long-Lived Keys
- Amazon Athena for Coding Agents: Query Tool Audit Logs and Eval Traces on S3 Without a Warehouse
- AWS Batch for Coding Agents: Parallel Overnight PR Reviews and Long-Running Jobs Without Lambda Timeouts
- Amazon Bedrock Prompt Management: Version and A/B Test Coding-Agent System Prompts
Deep-dive PDF
Get the expanded guide for this post — extra diagrams-style checklists, failure modes, and a production walkthrough. Free when you subscribe to CheatCoders.
Already subscribed? or open the subscribe page.
Discover more from CheatCoders
Subscribe to get the latest posts sent to your email.
