Day 77: Caching RAG Answers Safely

Caching RAG answers is a sharp knife. Done right, identical internal questions stop burning tokens. Done wrong, Tenant A sees Tenant B’s answer, or yesterday’s wrong runbook ships forever after a git push.

⚡ TL;DR: Cache only normalized question + tenant + corpus version + model/prompt version. Short TTLs. Never cache across tenants. Invalidate on index refresh. Prefer caching retrieval sets over final prose when answers must cite live code.

Safe cache key design

# ✅ Tenant-scoped, versioned cache key
import hashlib, json

def rag_cache_key(*, tenant_id: str, question: str, corpus_ver: str,
                  prompt_ver: str, model_id: str) -> str:
    norm = " ".join(question.lower().split())
    payload = {
        "t": tenant_id,
        "q": norm,
        "c": corpus_ver,   # git SHA or KB ingestion id
        "p": prompt_ver,
        "m": model_id,
    }
    raw = json.dumps(payload, sort_keys=True).encode()
    return "rag:" + hashlib.sha256(raw).hexdigest()

❌ Keying only on the question string. That is how you leak between customers and environments.

What to cache

For coding assistants, prefer caching ranked chunk IDs and re-running generation with fresh system rules. Prose caches go stale when POLICY.md changes.

Invalidation that actually runs

Hook your indexer (Day 27) to bump corpus_ver and delete keys by prefix or generation tag:

# ✅ Generation-based invalidation
def bump_corpus(tenant_id: str) -> str:
    ver = new_version()  # monotonic or git SHA
    redis.set(f"corpus_ver:{tenant_id}", ver)
    # optional: redis.scan_iter(f"rag:{tenant_id}:*") and delete
    return ver

Failure modes

Cross-env cache hits (staging key reused in prod) are as bad as cross-tenant. Include env in the key. Clock-skewed TTLs across gateway replicas can resurrect deleted answers — prefer generation counters over wall-clock alone when corpus_ver is unchanged but policy prompts changed.

Closing checklist

• [ ] Tenant id mandatory in every cache key
• [ ] corpus_ver + prompt_ver + model_id in key
• [ ] TTL ≤ your freshness SLO (often hours, not weeks)
• [ ] Integration test: same question, two tenants → two entries
• [ ] Never serve cached answer without citations still present in index

Series navigation

Day 76: p99 of Agents: Queueing, Not Just Model Latency · Day 78: Load Shedding When the Model Is Sick

Last updated September 11, 2026