RAG From Scratch: Chunking, Embeddings, Reranking and Evaluation (Real Numbers)

I asked a local model a simple question about a product called Harbor: what happens if an admin types the wrong password too many times? It answered with total confidence, named a config file called harbor.yml, and quoted the exact setting. None of it was true. The real documentation says admin accounts are never locked; Harbor blocks the IP address for an hour instead. There is no harbor.yml.

That is the problem retrieval augmented generation (RAG) is meant to fix. In this tutorial we build RAG from scratch in Node.js 22 with a fully local stack (Ollama, qwen2.5 3B and 7B, and nomic-embed-text embeddings), then measure seven chunking strategies on the same 32 questions, add reranking, assemble prompts with citations, and evaluate both retrieval and answers. No framework, no cloud, and every number below comes from a real run.

⚡ TL;DR – Chunking decides retrieval quality. Fixed 300-character chunks found the answer at rank 1 for 21/32 questions; markdown-aware chunks with the heading path on top hit 31/32 (MRR 0.969). – The “smart” semantic chunker (27/32) did not beat plain recursive splitting (28/32). – Reranking with a 3B judge made things worse (28 → 27). A 7B judge lifted recursive chunks from 28 to 31/32, MRR 0.984, at 8.4 s per question on CPU. – A 97% retrieval score still produced two wrong answers. Evaluate answers, not just retrieval.

Before you start

• Node.js 22 (for built-in fetch and top-level await).
• Ollama installed and running on localhost:11434.
• Models pulled: ollama pull nomic-embed-text, ollama pull qwen2.5:3b, ollama pull qwen2.5:7b.
• A folder of markdown docs. The video uses six fake help-center pages (900 to 1,400 characters each) for a made-up file-sync product called Harbor, with headings, code blocks and a table, because real docs look like that.
• Comfort with async JavaScript and arrays. No vector database is required.

Step 1: Understand what RAG actually does

RAG has four moves:

1. Cut your documents into chunks.
2. Turn every chunk into a list of numbers, an embedding.
3. When a question arrives, embed it too and find the closest chunks.
4. Paste those chunks into the prompt and let the model answer from what it just read.

Nothing is trained. The model does not learn your docs; it reads the right page before answering. That is why teams reach for RAG instead of fine-tuning: docs change and re-indexing takes seconds, you can show exactly where an answer came from, and it works with a small model on your own machine.

The catch is the whole point of this article: if retrieval picks the wrong chunk, the answer is wrong, and retrieval depends mostly on how you cut the documents.

Step 2: Build an evaluation set first

Before writing a single chunker, write the questions. Each question records the source file and a short fact that must appear in the retrieved text. Word the questions differently from the docs on purpose (for example “how many backup codes?” when the doc says “recovery codes”), because real users do not quote your docs.

// questions.mjs
export const questions = [
  { q: "What happens if an admin types the wrong password too many times?",
    file: "security.md", fact: "blocks the IP address for one hour" },
  { q: "How many backup codes do I get?",
    file: "security.md", fact: "10 recovery codes" },
  { q: "Can I get my money back?",
    file: "billing.md", fact: "refund" },
  // ... 32 in total
];

Step 3: Load documents with metadata

Loading is boring, but if you lose where a chunk came from here, you cannot cite it at the end.

// load.mjs
import { readdir, readFile } from "node:fs/promises";
import path from "node:path";

export async function loadDocs(dir) {
  const files = (await readdir(dir)).filter(f => f.endsWith(".md"));
  return Promise.all(files.map(async file => {
    const text = await readFile(path.join(dir, file), "utf8");
    const title = (text.match(/^#\s+(.+)$/m) || [, file])[1];
    return { file, title, text };
  }));
}

Step 4: Implement the seven chunkers

Why chunk at all? One embedding for a whole page is a blurry average of everything on it. Small chunks match sharply but lose context; big chunks keep context but flood the prompt with noise. Chunking is deciding where to cut.

1. Fixed size. Cut every 300 characters, ignoring words and sentences. This is where many tutorials start, and it literally cut one answer in half across two chunks in the test.

export const fixed = (t, size = 300) =>
  Array.from({ length: Math.ceil(t.length / size) }, (_, i) => t.slice(i * size, (i + 1) * size));

2. Fixed size with overlap. Same 300 characters, but each chunk repeats the last 75 of the previous one, so a sentence cut at the end of one chunk is usually whole at the start of the next. It costs more chunks and some duplicated text.

export function overlap(t, size = 300, ov = 75) {
  const out = [];
  for (let i = 0; i < t.length; i += size - ov) out.push(t.slice(i, i + size));
  return out;
}

3. Recursive splitting. The idea behind LangChain’s recursive character splitter, written from scratch: split on blank lines first so paragraphs stay together, then newlines, then sentence ends, then spaces. Cut a word only as a last resort.

const SEPS = ["\n\n", "\n", ". ", " "];
export function recursive(t, max = 400, seps = SEPS) {
  if (t.length <= max) return [t];
  const [sep, ...rest] = seps;
  if (!sep) return fixed(t, max);
  const out = []; let buf = "";
  for (const part of t.split(sep)) {
    const next = buf ? buf + sep + part : part;
    if (next.length <= max) { buf = next; continue; }
    if (buf) out.push(buf);
    if (part.length > max) { out.push(...recursive(part, max, rest)); buf = ""; }
    else buf = part;
  }
  if (buf) out.push(buf);
  return out;
}

4. Sentence packing. Split into sentences with a simple regex and pack whole sentences until a chunk is full. You never cut a sentence, but a naive regex flattens newlines, so tables and code get mashed together.

5. Semantic chunking. Embed every sentence and start a new chunk when neighbouring sentences stop being similar. It sounds clever and it is the slowest, because you embed every sentence up front. In the test it missed a topic boundary by one sentence and glued the “Conflicts” heading onto the end of a block-size paragraph.

6. Markdown-aware (structure-aware). Split on headings, keep code blocks and tables whole, and put the heading path on top of every chunk, such as Security > Login protection. Even a short chunk now carries what it is about. The same idea works for code: split on functions and classes, not characters.

export function markdownAware(doc) {
  const chunks = []; const path = []; let body = [];
  const flush = () => { if (body.join("").trim())
    chunks.push(`${doc.title} > ${path.join(" > ")}\n${body.join("\n").trim()}`); body = []; };
  for (const line of doc.text.split("\n")) {
    const h = line.match(/^(#{2,4})\s+(.+)/);
    if (h) { flush(); path.length = h[1].length - 2; path.push(h[2]); }
    else body.push(line);
  }
  flush();
  return chunks;
}

7. Parent-child. Search with small chunks (around one sentence) because they match sharply, but when one matches, hand the model the whole section it came from. Small to search, big to read.

Step 5: Embed with nomic-embed-text and cosine similarity

One thing people miss with nomic-embed-text: it wants task prefixes. Use search_document: for chunks and search_query: for questions. Cache every vector on disk so re-running the comparison is instant.

// embed.mjs
const cache = new Map();
export async function embed(text) {
  if (cache.has(text)) return cache.get(text);
  const r = await fetch("http://localhost:11434/api/embed", {
    method: "POST",
    body: JSON.stringify({ model: "nomic-embed-text", input: text }),
  });
  const v = (await r.json()).embeddings[0];   // 768 numbers
  cache.set(text, v);
  return v;
}
export function cosine(a, b) {
  let dot = 0, na = 0, nb = 0;
  for (let i = 0; i < a.length; i++) { dot += a[i] * b[i]; na += a[i] ** 2; nb += b[i] ** 2; }
  return dot / Math.sqrt(na * nb);
}

A real run shows why retrieval is ranking, not knowing. For “Can I get my money back?”, the refund sentence scored 0.66, a session-expiry sentence scored 0.46, and a yearly-discount sentence scored 0.54: close to the right answer simply because it is also about money and yearly plans.

Step 6: Build a vector store in 15 lines

For a few thousand chunks you do not need a database. It is an array and a loop. Real vector databases do the same thing with an index so it scales.

// store.mjs
import { embed, cosine } from "./embed.mjs";
export class Store {
  items = [];
  async add(text, meta) {
    this.items.push({ text, meta, v: await embed("search_document: " + text) });
  }
  async search(question, k = 5) {
    const qv = await embed("search_query: " + question);
    return this.items
      .map(it => ({ ...it, score: cosine(qv, it.v) }))
      .sort((a, b) => b.score - a.score)
      .slice(0, k);
  }
}

Step 7: Compare the chunkers fairly

Same six docs, same 32 questions, same embedding model. For each strategy, build a store, take the top 5 for every question, and record the rank of the first chunk containing the expected fact.

• hit@1: the answer was in the very first chunk.
• MRR (mean reciprocal rank): 1 point for rank 1, 1/2 for rank 2, 1/3 for rank 3, and so on, averaged.
• Also log how many characters the top 3 put in the prompt, because more text is not free.

let hit1 = 0, rr = 0;
for (const { q, fact } of questions) {
  const res = await store.search(q, 5);
  const rank = res.findIndex(r => r.text.toLowerCase().includes(fact.toLowerCase())) + 1;
  if (rank === 1) hit1++;
  if (rank) rr += 1 / rank;
}
console.log(name, `${hit1}/32`, (rr / questions.length).toFixed(3));

Real results (answer found at rank 1, out of 32):

Fixed size also never found 3 answers at all: the read-only rule, the manager role and a storage-quota sentence were each split across two chunks. Recursive splitting reached 28 with the smallest prompts, and semantic chunking produced the biggest prompts without beating it. The winner was the boring option: respect the structure the writer already gave you.

Markdown-aware still had one miss nobody else had. “Where does the first admin password appear?” lives right after a bash code block that was kept whole, so the chunk’s vector was mostly about curl and chmod. Keeping code blocks whole is right, but the sentence after a code block should probably live with its own paragraph.

Parent-child scored 28 because two sentences from the same short section often both matched, so one of the top-5 slots was usually the same parent twice. Deduplicating did not change the score. On long documents it has much more room to help.

Honest caveat: this is 32 questions over six short pages, so one question of difference is noise. Fixed size being 10 behind is not. The takeaway is the method: write questions, measure, then decide.

Step 8: Add reranking with a local LLM judge

Vector search is fast but rough. Reranking grabs 10 candidates, asks a model to score each against the question from 0 to 10, and reorders them. A proper cross-encoder would be faster; this version needs only Ollama.

async function judge(model, q, chunk) {
  const r = await fetch("http://localhost:11434/api/generate", {
    method: "POST",
    body: JSON.stringify({ model, stream: false, options: { temperature: 0 },
      prompt: `Question: ${q}\nPassage: ${chunk}\nHow well does the passage answer the question? Reply with one integer 0-10.` }),
  });
  return parseInt((await r.json()).response, 10) || 0;
}
export async function rerank(model, q, candidates) {
  const scored = [];
  for (const c of candidates) scored.push({ ...c, j: await judge(model, q, c.text) });
  return scored.sort((a, b) => b.j - a.j);
}

Real results on recursive chunks:

• qwen2.5:3b as judge: 28 → 27 at rank 1. Worse, and 4.5 s per question.
• qwen2.5:7b as judge: 28 → 31, MRR 0.984, as good as the best chunker, at 8.4 s per question on CPU.
• 3B judge on fixed-size chunks: 21 before, 21 after. A reranker can only reorder what retrieval found; it cannot put back half a sentence.

Step 9: Assemble the prompt with citations

Take the top 3 markdown-aware chunks, number them, add the source file, and instruct the model to answer only from them, cite like [1], and say “I don’t know” when the answer is missing.

const context = top3.map((c, i) => `[${i + 1}] (${c.meta.file})\n${c.text}`).join("\n\n");
const prompt = `Answer only from the sources below. Cite like [1].
First find the sentence that answers the question, even if it uses different words.
Only say "I don't know" if nothing is related.

${context}

Question: ${q}`;

What happened in real runs: retrieval put the login-protection section at position 1, yet the 3B model first said “I don’t know” because the doc says “failed login attempts”, not “wrong password”. After adding the “find the sentence first” line, 3B answered with a citation, and was wrong: it said admins get locked for 30 minutes, which is the rule for normal users; the very next sentence says admins are never locked. An out-of-scope SAML question correctly got “I don’t know”. And the same prompt change made the 7B model claim monthly plans can be refunded. They cannot. The fix for one question broke another.

Step 10: Evaluate retrieval and answers separately

Evaluation has two halves. Retrieval (hit rate, MRR) is cheap and fast to compute. Answers must be checked against the expected fact. A 97% retrieval score still produced two different wrong answers to the same question. Include questions that are not in the docs, and re-run everything after every change.

Troubleshooting

• Scores look random: you forgot the search_document: / search_query: prefixes for nomic-embed-text.
• Some answers are never retrieved: print the chunks containing the fact. If the sentence is split, add overlap or switch to recursive or structure-aware chunking.
• A short sentence never matches: it is probably glued to a code block or table that dominates the vector. Give it its own chunk or prepend the heading path.
• Reranking hurts: your judge model is too weak. Measure before and after; drop reranking or use a stronger judge or a cross-encoder.
• The model says “I don’t know” with the answer in context: the wording differs. Ask it to locate the supporting sentence first, then re-test every other question.
• Re-runs are slow: cache embeddings on disk keyed by the exact text.

Best practices

• Prefer structure-aware chunks with the heading path on every chunk.
• For docs with no structure, use recursive splitting with some overlap.
• Write around 30 real questions before tuning anything, including some with no answer in the docs.
• Only rerank with a model strong enough to help, and time it.
• Keep file names and titles with every chunk so citations are possible.
• Test answers, not only retrieval, and re-run the whole set after every prompt change.

FAQ

Is semantic chunking worth it?

Not on these docs. It was the slowest strategy, produced the biggest prompts, and scored 27/32 versus 28/32 for plain recursive splitting. Measure on your own data before paying for it.

Do I need a vector database?

Not for a few thousand chunks. An in-memory array with cosine similarity is fine. Reach for a database with an approximate-nearest-neighbour index when the corpus or query volume grows.

Why not fine-tune the model on my docs instead?

Docs change, re-indexing takes seconds, RAG can cite sources, and it works with small local models. Fine-tuning changes style and behaviour better than it injects fresh facts.

Does reranking always help?

No. A 3B judge lowered hit@1 from 28 to 27; a 7B judge raised it to 31 but cost 8.4 s per question on CPU.

Key takeaways

• Retrieval quality is mostly a chunking decision. Fixed 300-character chunks lost 10 questions versus the best strategy.
• Markdown-aware chunks with heading paths won: 31/32, MRR 0.969.
• Fancy is not automatically better: semantic chunking did not beat recursive splitting.
• A reranker can only reorder what retrieval found, and a weak judge makes results worse.
• Good retrieval does not guarantee good answers. Evaluate both halves and re-run after every change.