Multi-Index RAG Design: Separate Code, Tickets, and Runbook Corpora

One giant vector index for code + Jira + Notion + PagerDuty runbooks produces confidently wrong blends: a stack-trace question returns an outdated epic comment, an implementation question returns a severity definition. Senior design keeps separate corpora with an intentional router that picks indexes (and prompts) per intent.

⚡ TL;DR: Split indexes: code, tickets, runbooks, adrs; classify intent with a cheap model + heuristics; retrieve only from selected corpora; fuse answers with corpus tags in citations. Don’t “just raise k” on a mixed pile. See RAG over ADRs and KB metadata filters.

Why mixing fails

Embeddings from heterogeneous domains collide. Separate ANN graphs (or Bedrock KBs) preserve geometry per domain.

Router before retrieve

type Corpus = "code" | "tickets" | "runbooks" | "adrs";

type Route = { corpora: Corpus[]; reason: string };

function heuristicRoute(q: string): Route | null {
  if (/runbook|pagerduty|sev[1-3]|on-?call|rollback/i.test(q))
    return { corpora: ["runbooks", "adrs"], reason: "ops_keywords" };
  if (/jira|ticket|sprint|ac \/|acceptance/i.test(q))
    return { corpora: ["tickets", "code"], reason: "ticket_keywords" };
  if (/why did we|adr|decision/i.test(q))
    return { corpora: ["adrs", "code"], reason: "decision" };
  return null;
}

async function route(q: string): Promise<Route> {
  return heuristicRoute(q) ?? (await llmRoute(q)); // nova-lite JSON
}

async function multiRetrieve(q: string) {
  const r = await route(q);
  const hits = await Promise.all(
    r.corpora.map(async (c) => ({
      corpus: c,
      chunks: await indexes[c].search(q, { k: c === "code" ? 8 : 5 }),
    })),
  );
  return { route: r, hits };
}

Default code-only for IDE sidebar; add corpora explicitly for “Ask ops” / “Ask tickets” modes. Do not silent all-corpora retrieve for every chat box in the company.

Citation tags by corpus

{
  "claims": [
    {
      "text": "Rollback is feature-flag flip then drain.",
      "path": "runbooks/payments/rollback.md",
      "corpus": "runbooks",
      "start_line": 12,
      "end_line": 31
    }
  ]
}

UI badges (code / runbook / ticket) stop engineers from treating a Jira opinion as source code.

Lifecycle and permissions

• Code: git webhook sync; engineers’ IAM.
• Tickets: scrub injection (sanitize Jira); stricter ACLs.
• Runbooks: owned by SRE; change control; pin current version in citations.
• ADRs: slow-changing; high authority weight in fusion.

Fusion policy (illustrative):
  runbooks > adrs > code > tickets   for on-call intents
  code > adrs > tickets > runbooks   for implementation intents

Closing checklist

Dos
– Separate indexes/KBs per corpus
– Route with heuristics + cheap LLM; log route decisions
– Tag citations with corpus
– Apply corpus-specific IAM and sanitization
– Eval per route (ops vs implementation slices)

Donts
– Do not dump Notion + git into one OpenSearch index “for simplicity”
– Do not let ticket text outrank runbooks during incidents
– Do not skip ACLs because “it’s just embeddings”
– Do not use one prompt template for all corpora
– Do not hide which corpus an answer used

Related reading

• Bedrock Knowledge Base Metadata Filters
• RAG over ADRs
• Prompt Injection in Trackers
• Hallucination Triage for RAG

Last updated on September 11, 2026