19 views
One giant vector index for code + Jira + Notion + PagerDuty runbooks produces confidently wrong blends: a stack-trace question returns an outdated epic comment, an implementation question returns a severity definition. Senior design keeps separate corpora with an intentional router that picks indexes (and prompts) per intent.
⚡ TL;DR: Split indexes:code,tickets,runbooks,adrs; classify intent with a cheap model + heuristics; retrieve only from selected corpora; fuse answers with corpus tags in citations. Don’t “just raise k” on a mixed pile. See RAG over ADRs and KB metadata filters.
Multi-Index RAG Design: production guidance
| Mixed retrieval symptom | Cause |
|---|---|
| Runbook steps cite random PR descriptions | Ticket text dominates BM25 |
| “How is auth implemented?” → incident timeline | Semantic neighbor across corpora |
| Conflicting TTLs / ownership | Different truth sources, one ranker |
Router before retrieve
typescript
Default code-only for IDE sidebar; add corpora explicitly for “Ask ops” / “Ask tickets” modes. Do not silent all-corpora retrieve for every chat box in the company.
type Corpus = "code" | "tickets" | "runbooks" | "adrs";
type Route = { corpora: Corpus[]; reason: string };
function heuristicRoute(q: string): Route | null {
if (/runbook|pagerduty|sev[1-3]|on-?call|rollback/i.test(q))
return { corpora: ["runbooks", "adrs"], reason: "ops_keywords" };
if (/jira|ticket|sprint|ac \/|acceptance/i.test(q))
return { corpora: ["tickets", "code"], reason: "ticket_keywords" };
if (/why did we|adr|decision/i.test(q))
return { corpora: ["adrs", "code"], reason: "decision" };
return null;
}
async function route(q: string): Promise<Route> {
return heuristicRoute(q) ?? (await llmRoute(q)); // nova-lite JSON
}
async function multiRetrieve(q: string) {
const r = await route(q);
const hits = await Promise.all(
r.corpora.map(async (c) => ({
corpus: c,
chunks: await indexes[c].search(q, { k: c === "code" ? 8 : 5 }),
})),
);
return { route: r, hits };
}
Citation tags by corpus
json
UI badges ({
"claims": [
{
"text": "Rollback is feature-flag flip then drain.",
"path": "runbooks/payments/rollback.md",
"corpus": "runbooks",
"start_line": 12,
"end_line": 31
}
]
}
code / runbook / ticket) stop engineers from treating a Jira opinion as source code.
Lifecycle and permissions
- Code: git webhook sync; engineers’ IAM.
- Tickets: scrub injection (sanitize Jira); stricter ACLs.
- Runbooks: owned by SRE; change control; pin current version in citations.
- ADRs: slow-changing; high authority weight in fusion.
text
Fusion policy (illustrative):
runbooks > adrs > code > tickets for on-call intents
code > adrs > tickets > runbooks for implementation intents
Closing checklist
Dos – Separate indexes/KBs per corpus – Route with heuristics + cheap LLM; log route decisions – Tag citations with corpus – Apply corpus-specific IAM and sanitization – Eval per route (ops vs implementation slices) Donts – Do not dump Notion + git into one OpenSearch index “for simplicity” – Do not let ticket text outrank runbooks during incidents – Do not skip ACLs because “it’s just embeddings” – Do not use one prompt template for all corpora – Do not hide which corpus an answer usedRelated reading
- Bedrock Knowledge Base Metadata Filters
- RAG over ADRs
- Prompt Injection in Trackers
- Hallucination Triage for RAG
Watch: 100 Days of AI on YouTube
Short videos from CheatCoders, one AI engineering topic per day.
Most viewed
- Python Decorators Explained: From Simple Wrappers to Production Patterns
- Claude Code vs Cursor vs GitHub Copilot: Choose by Workflow, and Set Up One Repo That Works With All Three
- AI Agent Frameworks in 2025: LangGraph vs CrewAI vs AutoGen vs Raw API
- Python String Methods: Every str Method With Real Production Examples
- Css Animation Performance: Performance-Optimized Motion for Modern UIs
Newly added
- AWS Systems Manager Parameter Store: Hierarchical Runtime Config for Coding Agents
- Amazon Cognito Identity Pools: Federate Coding-Agent Tool Callers Without Long-Lived Keys
- Amazon Athena for Coding Agents: Query Tool Audit Logs and Eval Traces on S3 Without a Warehouse
- AWS Batch for Coding Agents: Parallel Overnight PR Reviews and Long-Running Jobs Without Lambda Timeouts
- Amazon Bedrock Prompt Management: Version and A/B Test Coding-Agent System Prompts
Deep-dive PDF
Get the expanded guide for this post — extra diagrams-style checklists, failure modes, and a production walkthrough. Free when you subscribe to CheatCoders.
Already subscribed? or open the subscribe page.
Discover more from CheatCoders
Subscribe to get the latest posts sent to your email.
