Day 28: Multimodal RAG: Diagrams, PDFs, and Screenshots

Dumping every PDF page through OCR and embedding the sludge is how you retrieve cookie banners instead of sequence diagrams. Multimodal RAG is a triage problem: OCR versus structured caption versus native image embedding — chosen per asset class, tagged in metadata, and filtered at query time.

⚡ TL;DR: OCR text-heavy PDFs with layout awareness; caption architecture diagrams into nodes/edges JSON; keep screenshots as images plus short captions for multimodal models. Store modality on every chunk. Include diagram/PDF items in your eval pack.

Triage before you embed

Blind OCR on a PNG sequence diagram yields broken arrows and no edges. Structured captions preserve topology that GraphRAG (Day 25) and plain hybrid search can use.

Caption diagrams as graphs

CAPTION_SCHEMA = {
  "type": "object",
  "required": ["title", "nodes", "edges", "summary"],
  "properties": {
    "title": {"type": "string"},
    "nodes": {"type": "array", "items": {"type": "string"}},
    "edges": {"type": "array", "items": {
      "type": "object",
      "required": ["from", "to"],
      "properties": {
        "from": {"type": "string"},
        "to": {"type": "string"},
        "label": {"type": "string"}
      }
    }},
    "summary": {"type": "string"}
  },
  "additionalProperties": False,
}

PROMPT = """Describe this architecture diagram as JSON matching the schema.
Prefer service names over colors. Ignore logos and watermarks."""

text = tesseract(png)
embed(text)  # retrieval lottery

Human-review the first caption for each critical diagram. Automate thereafter with spot checks.

PDFs: layout-aware chunking

Chunk by section headings and blocks, not fixed 512-token windows that slice across columns.

import fitz  # PyMuPDF

def section_chunks(path: str):
    doc = fitz.open(path)
    for page in doc:
        for b in page.get_text("dict")["blocks"]:
            if b.get("type") != 0:
                continue
            text = "\n".join(span["text"] for line in b["lines"] for span in line["spans"])
            if len(text.strip()) < 40:
                continue
            yield {"page": page.number + 1, "text": text.strip(), "modality": "pdf_text"}

Query-time modality filters

If the user asks for a sequence diagram, boost modality=diagram. If they paste an exact error string, prefer modality=text. Multimodal generation can attach image bytes for cited diagrams while still failing closed on text claims (Day 26).

Production checklist

• [ ] Ingest tags modality + source_asset_id
• [ ] Diagrams get structured captions (reviewed on first index)
• [ ] Screenshots never enter without a caption
• [ ] Citations can return image URLs
• [ ] Eval set includes ≥10 diagram/PDF questions
• [ ] OCR quality sampled weekly on new asset types

Series navigation

← Day 27 · Day 29 →

Last updated September 11, 2026