Day 6: Chunking Strategies for Code, Tickets, and Runbooks

Chunking is where most RAG systems quietly die. Too small and you lose the decision that lived in the next paragraph; too large and retrieval returns a blob the generator cannot cite precisely. Day 6 treats chunking as a corpus-specific engineering choice — fixed windows, recursive splitters, AST-aware code cuts, and late chunking — measured against your tickets and runbooks.

⚡ TL;DR: Default recursive markdown/text splitters are a starting point, not a religion. Use AST or symbol-aware cuts for code. Keep ticket threads as conversation-aware slices. Prefer late chunking when you need document-level context in embeddings. Evaluate with retrieval metrics, not aesthetics.

Fixed windows: honest and limited

Fixed token windows (e.g. 512 with 64 overlap) are predictable for cost and index size. They shred tables, numbered runbook steps, and functions mid-body.

# ✅ Use fixed windows only when structure is weak (plain logs, scraped text)
def fixed_chunks(text: str, size=512, overlap=64, encode=len_tokens):
    toks = encode(text)
    out, i = [], 0
    while i < len(toks):
        out.append(toks[i:i+size])
        i += max(1, size - overlap)
    return out

❌ Pasting chunk_size=1000 from a tutorial into a Terraform monorepo and declaring RAG “done.”

Overlap helps with boundary bleed but does not restore hierarchy. Always store parent_doc_id, section_path, and token_count on each chunk for debugging.

Recursive and structure-aware splitters

Recursive character/markdown splitters walk a separator priority (\n##, \n, ) trying to keep headings with their bodies. For runbooks written in Markdown this is often the right first pass.

Enhance it:

• Never split inside fenced code blocks.
• Keep “Symptoms / Impact / Mitigation” sections atomic when possible.
• Cap max tokens; if a section is huge, split on numbered steps secondarily.

For HTML Confluence exports, normalize to Markdown before chunking or you will index navigation chrome.

AST and symbol chunking for code

Code wants symbol boundaries: function, class, Terraform resource, Kubernetes manifest document (---).

# ✅ Sketch: chunk by top-level symbols, attach file path + symbol name
import ast

def py_chunks(path: str, src: str) -> list[dict]:
    tree = ast.parse(src)
    lines = src.splitlines()
    chunks = []
    for node in tree.body:
        if isinstance(node, (ast.FunctionDef, ast.AsyncFunctionDef, ast.ClassDef)):
            text = "\n".join(lines[node.lineno-1:node.end_lineno])
            chunks.append({
                "id": f"{path}::{node.name}",
                "text": text,
                "symbol": node.name,
                "kind": type(node).__name__,
            })
    return chunks

Include a short file-level summary chunk (imports + module docstring) so “where is auth middleware?” can retrieve the module, not only a leaf function. For multi-language repos, per-language strategies beat one global splitter.

Tickets, threads, and late chunking

Tickets are dialogues. Chunk by comment groups or “description + last N comments,” and embed issue keys in metadata for hybrid search. Do not merge unrelated tickets into one mega-chunk.

Late chunking embeds long documents with contextualized token representations, then pools spans — improving retrieval when early chunks need document context. Use it when your eval shows standard independent chunk embeddings miss cross-section references (“see Prerequisites above”). It is not free; measure latency and cost.

How to pick (decision table)

Run A/B chunkers through the Day 3/9 eval set. The winner is the one that lifts Recall@10 on your queries.

Closing checklist

• [ ] Document chunker choice per corpus type in the repo README
• [ ] Preserve code fences and tables; store section paths
• [ ] Use AST/symbol chunking for source code
• [ ] Keep ticket IDs in metadata; chunk threads deliberately
• [ ] A/B chunk strategies on a labeled retrieval set before locking
• [ ] Re-chunk and re-embed when templates change, not only when models change

Worked example: Terraform chunking

Index each resource / data / module block as its own chunk with address metadata (aws_iam_role.api). Add a file-level chunk listing addresses. Queries like “who can assume the API role?” then hit the role block without dragging unrelated aws_s3_bucket noise.

# Keep comments above the resource inside the same chunk — they carry intent

Re-run retrieval eval after changing splitters; cosmetic Markdown prettiness is irrelevant.

Failure modes to watch

• Splitting inside YAML list items for K8s manifests.
• Huge README chunks drowning symbol search.
• Ticket mega-chunks mixing three incidents.
• No re-embed after splitter change.

Field notes from production

Confluence and Notion exporters inject boilerplate (‘Skip to main content’) that becomes high-frequency junk neighbors. Strip chrome in ingest. For PDFs, table-as-chunk beats paragraph slicing when operators ask about numeric thresholds in SLOs.

Implementation sketch

# Implementation sketch: skip splitting inside fences
def split_markdown(md: str):
    parts, buf, in_fence = [], [], False
    for line in md.splitlines():
        if line.strip().startswith("```"):
            in_fence = not in_fence
        buf.append(line)
        if not in_fence and line.startswith("## "):
            parts.append("\n".join(buf[:-1])); buf = [line]
    parts.append("\n".join(buf)); return [p for p in parts if p.strip()]

Operator addendum

When you change chunkers, keep the old index readable until evals pass on the new alias. Dual-write embeddings for a soak period if you cannot afford a big-bang reindex overnight.

Series navigation

← Day 5 · Day 7 →

Last updated September 11, 2026