Day 5: Your First RAG Pipeline That You Can Debug

RAG fails quietly: wrong chunk, empty retrieval, fluent hallucination with a confident tone. Day 5 ships a pipeline you can debug — every hop carries a trace ID, every knowledge answer carries citations, and you have a written policy for low-confidence retrieval instead of improvisation.

⚡ TL;DR: Trace ingest → chunk → embed → retrieve → generate. Require citations mapped to chunk IDs. If retrieval scores or hit-rate are below threshold, refuse or clarify — do not let the model freestyle operational guidance.

Four stages and their failure modes

1. Ingest — source ACLs, versioning, PII scrubbing, canonical URLs
2. Chunk — boundaries that preserve meaning (deep dive on Day 6)
3. Retrieve — metadata filters + ANN/BM25 with logged scores
4. Generate — answer from evidence only; cite or refuse

# ✅ Minimal debuggable skeleton
from dataclasses import dataclass, field

@dataclass
class Trace:
    request_id: str
    tenant: str
    chunks_retrieved: list[str] = field(default_factory=list)
    scores: list[float] = field(default_factory=list)
    cited: list[str] = field(default_factory=list)
    refused: bool = False

MIN_SCORE = 0.25  # calibrate on your eval set — not a universal constant

def answer(query: str, tenant: str, trace: Trace) -> str:
    hits = retrieve(query, tenant=tenant, k=8)
    trace.chunks_retrieved = [h.id for h in hits]
    trace.scores = [h.score for h in hits]
    if not hits or hits[0].score < MIN_SCORE:
        trace.refused = True
        return (
            "I do not have grounded evidence for that in your docs. "
            "Share a link, ticket ID, or service name."
        )
    text, cited = generate_with_citations(query, format_evidence(hits))
    trace.cited = cited
    if set(cited) - set(trace.chunks_retrieved):
        raise RuntimeError("model cited unknown chunk ids")
    return text

Emit trace to OpenTelemetry/CloudWatch Logs with model ID, embedding model, and index alias. When a user says “it hallucinated,” you must see whether retrieve or generate failed.

Citations that mean something

A citation is a chunk ID (or repo path + line span), not a decorative footnote. The UI should deep-link to the source document version that was embedded. If the model cannot cite for a factual/ops question, treat that as a failed generation.

✅ "Multi-AZ RDS failover is typically under ~60s for the app to recover [chunk:rds-runbook-12]."
❌ "Failover is usually quick."   // uncited operational claim — reject

Add a validator that checks every [chunk:…] against the retrieved set before the response leaves your API Gateway.

Failure budget for RAG (SRE language)

Define error budgets: e.g. citation coverage ≥ 95% on knowledge intents; refusal rate for out-of-corpus queries is healthy, not a bug.

AWS reference shape (Bedrock + OpenSearch or pgvector)

A common production path: S3 (versioned docs) → Lambda ingest → embed on Bedrock → OpenSearch k-NN or Aurora pgvector → retrieve under IAM-scoped filters → generate on Bedrock with a cite-or-refuse system contract. Brand names change; the invariant is debuggability. Without traces you cannot tell a bad index from a bad prompt.

Pair this day with Day 9’s eval harness before you demo to leadership. A shiny chat UI without hit-rate charts is a liability.

Closing checklist

• [ ] End-to-end request_id across ingest, retrieve, and generate
• [ ] Persist retrieval IDs + scores with every answer
• [ ] Enforce cite-or-refuse for factual and ops questions
• [ ] Calibrate MIN_SCORE on a labeled set; document it
• [ ] Dashboard refusal rate, citation coverage, and p95 latency
• [ ] On-call runbook: how to distinguish retrieve vs generate faults

Worked example: one request_id across the hop

Propagate X-Request-Id from API Gateway through retrieve and generate. In CloudWatch Logs Insights, query that ID and you should see: filters applied, top chunk IDs, scores, model ID, refusal boolean, citation list. If any field is missing, your “debuggable RAG” claim is false.

logger.info("rag_turn", extra={
  "request_id": rid,
  "tenant": tenant,
  "scores": scores[:8],
  "cited": cited,
  "refused": refused,
  "model": model_id,
})

Failure modes to watch

• Citations to chunks never retrieved (validator missing).
• MIN_SCORE cargo-culted from another corpus.
• Tracing only the generate call, blind on retrieve.
• Demo scripts that never ask out-of-corpus questions.

Field notes from production

Separate dashboards for knowledge intents vs chitchat. Citation coverage on chitchat is meaningless. Classify intent first (small model / rules) then apply cite-or-refuse only where it matters. This also saves tokens (Day 15) by skipping retrieval on greetings.

Implementation sketch

# Implementation sketch: citation gate
def gate(answer, cited, retrieved):
    if not cited or set(cited) - set(retrieved):
        return Refuse("uncited_or_unknown_citation")
    return answer

Operator addendum

Teach support to read traces before paging ML. Ninety percent of ‘hallucinations’ are retrieve misses visible in scores. A one-page on-call card with CloudWatch Insights snippets pays for itself.

Separating retrieve bugs from generate bugs

Create two synthetic fixtures: (1) perfect retrieval pack with a known answer — if generate still hallucinates, fix prompts/validators; (2) empty retrieval — if generate answers anyway, your refuse path is broken. Run both in CI. Most “RAG quality” debates collapse once you know which stage failed. Put the stage name in the user-visible error for internal tools (“No evidence retrieved”) so support does not blame the model by default.

Series navigation

← Day 4 · Day 6 →

Last updated September 11, 2026