Cost-Aware RAG Caches: Reuse Answers for Identical Code Questions

Teams ask the same “where is rate limiting enforced?” twenty times a day. Without a cache keyed on normalized question + repo SHA, you pay Bedrock twenty times for one answer that drifts the moment main moves. Cost-aware RAG caches cut spend without serving ghosts from deleted modules.

⚡ TL;DR: Normalize question text, hash with tree SHA (or package digest), store answer + citations + model ID in Redis/Dynamo with TTL bounded by merge rate. Invalidate on path touches cited in the answer. Never cache across tenants. Pair with LLM Cost Controls and Bedrock Knowledge Base Sync.

Cache key design

import { createHash } from "node:crypto";

export function cacheKey(opts: {
  tenantId: string;
  repo: string;
  treeSha: string;
  question: string;
  modelId: string;
}) {
  const q = opts.question
    .toLowerCase()
    .replace(/[`*_]/g, "")
    .replace(/\s+/g, " ")
    .trim();
  const material = [opts.tenantId, opts.repo, opts.treeSha, opts.modelId, q].join("|");
  return createHash("sha256").update(material).digest("hex");
}

// ❌ Keying only on question → stale answers after refactors
// ❌ Omitting tenantId → cross-tenant leakage

DynamoDB item shape

type RagCacheItem = {
  pk: string; // tenant#repo
  sk: string; // key hash
  treeSha: string;
  answerMd: string;
  citations: { path: string; start: number; end: number }[];
  modelId: string;
  tokensSaved: number;
  expiresAt: number; // epoch seconds (TTL attribute)
};

On hit, increment tokensSaved metrics for FinOps dashboards (LLM Cost Controls).

Invalidation that tracks citations

export async function invalidateForPaths(repo: string, changed: string[]) {
  // Query GSIs of cached citations overlapping changed paths
  const items = await queryByCitedPaths(repo, changed);
  await Promise.all(items.map((i) => deleteItem(i)));
}

// Hook from git webhook after merge — same channel as KB sync

Webhook-driven freshness beats blind TTLs alone; see KB sync webhooks.

Stampede and single-flight

Monday standups produce identical questions. Use single-flight (one in-flight Bedrock call per key) so ten IDE clients do not fan out ten full RAG paths:

const inflight = new Map<string, Promise<RagAnswer>>();

export async function getOrRetrieve(key: string, produce: () => Promise<RagAnswer>) {
  const hit = await cache.get(key);
  if (hit) return hit;
  let p = inflight.get(key);
  if (!p) {
    p = produce().finally(() => inflight.delete(key));
    inflight.set(key, p);
  }
  const ans = await p;
  await cache.set(key, ans);
  return ans;
}

When not to cache

Closing checklist

• [ ] Key includes tenant, repo, tree SHA, model ID, normalized question
• [ ] TTL + citation-path invalidation on merge
• [ ] Metrics: hit rate, tokens saved, stale-serve incidents
• [ ] No cache for working-tree or incident-contextual prompts
• [ ] Encryption at rest; no secrets in cached answers (redact first)
• [ ] Load test cache stampede on Monday mornings

Related reading

• LLM Cost Controls: Token Budgets Per PR and Per Engineer
• Bedrock Knowledge Base Sync: Git Webhooks That Prevent Index Drift
• Prompt Caching Pitfalls: Stale Coding Rules After Major Repo Moves
• Bedrock Retrieval Filters: Tenant Isolation for Multi-SaaS Code RAG

Last updated on September 11, 2026