Chunk Overlap Evaluation: Precision Versus Continuity for Long Functions

Overlap is the quiet knob that makes or breaks code RAG. Too little and long functions lose the signature when the body starts a new window. Too much and your top-k fills with near-duplicates of the same method, crowding out the caller that actually answers the question. Seniors do not guess overlap — they measure precision, continuity, and duplicate rate on a frozen golden set.

⚡ TL;DR: Sweep overlap at 0 / 10% / 20% / 35% of chunk size on AST and fixed windows. Track hit-rate@k, unique-symbol coverage, and duplicate-in-top-k. Prefer symbol-bounded chunks with small header overlap over huge sliding windows. Pair with Bedrock Knowledge Bases chunking and Code RAG rerankers.

Define the metrics before you sweep

# eval/overlap_metrics.py
from dataclasses import dataclass

@dataclass
class OverlapReport:
    overlap_pct: float
    hit_at_5: float
    unique_symbols_in_topk: float
    duplicate_ratio: float  # 1 - unique/len(topk)
    continuity_score: float  # fraction of golds spanning adjacent chunks that both retrieve

def duplicate_ratio(ids: list[str]) -> float:
    if not ids:
        return 0.0
    # ✅ Normalize by symbol id so path::fn duplicates count
    base = [i.split("::")[0] + "::" + i.split("::")[-1] for i in ids]
    return 1.0 - (len(set(base)) / len(base))

❌ Tuning overlap by “feels contiguous in the IDE” — that optimizes for reading, not for retrieval ranking.

Sweep overlap on a fixed golden set

def sweep_overlaps(chunk_fn, queries, gold, sizes=(512, 768), overlaps=(0.0, 0.1, 0.2, 0.35)):
    rows = []
    for size in sizes:
        for ov in overlaps:
            chunks = chunk_fn(size=size, overlap=int(size * ov))
            index = build_temp_index(chunks)  # ephemeral OpenSearch / pgvector
            hits, dups, cont = [], [], []
            for q in queries:
                top = retrieve(index, q.text, k=5)
                hits.append(1.0 if gold[q.id] & {c.symbol for c in top} else 0.0)
                dups.append(duplicate_ratio([c.id for c in top]))
                cont.append(continuity(q, top, gold))
            rows.append({
                "size": size, "overlap": ov,
                "hit@5": sum(hits)/len(hits),
                "dup": sum(dups)/len(dups),
                "continuity": sum(cont)/len(cont),
            })
    return rows  # ✅ Pick knee: max hit@5 with dup < 0.25

Prefer header continuity over body duplication

For AST-aware units, do not slide a window through the function body. Instead keep the full symbol and prepend a thin continuity header: parent class, imports used, and previous sibling signature.

// chunk/continuity-header.ts
export function withContinuityHeader(symbol: {
  path: string; name: string; text: string; parent?: string; prevSibling?: string;
}) {
  const header = [
    `// file: ${symbol.path}`,
    symbol.parent ? `// class: ${symbol.parent}` : null,
    symbol.prevSibling ? `// prev: ${symbol.prevSibling}` : null,
  ].filter(Boolean).join("\n");
  // ✅ Continuity without overlapping the previous body into this chunk
  return `${header}\n${symbol.text}`;
}

Multi-file features need feature packs, not more overlap

When a question spans auth/session.ts and middleware/requireUser.ts, overlap inside one file will never surface the other. Build feature packs: co-chunk callers and callees discovered from a static call graph, then evaluate pack recall separately from single-symbol recall.

Closing checklist

✅ Dos
– ✅ Sweep overlap on a versioned golden query set
– ✅ Report hit@k, duplicate ratio, and continuity together
– ✅ Use AST chunks with thin continuity headers
– ✅ Cap duplicate-in-top-k (e.g. < 0.25) as a hard gate
– ✅ Evaluate multi-file packs separately from single-symbol recall

❌ Don’ts
– ❌ Don’t copy a blog’s “20% overlap” without measuring
– ❌ Don’t slide huge windows through already-complete functions
– ❌ Don’t ignore duplicates when hit@k looks “fine”
– ❌ Don’t conflate reading continuity with retrieval precision
– ❌ Don’t retune overlap every week without freezing goldens

Related reading

• Bedrock Knowledge Bases: Chunking Strategies That Fit Code RAG
• Code RAG Rerankers: Cross-Encoders After Cheap Vector Recall
• Hybrid Search for Code: BM25, Vectors, and Symbol Indexes Together
• RAG Evaluation on AWS: Hit Rate, Faithfulness, and Cost Curves

Last updated on September 11, 2026