RAG Reranking Explained: Fix Wrong Answers (JavaScript)

Your RAG chatbot passed every test last week. Then somebody added an archived changelog to the docs, a page with last year’s prices that even says it’s no longer current, and the bot started giving confident, cited, wrong answers. Nothing in the code changed. Retrieval just found a chunk that looked relevant and the model trusted it.

Reranking is the usual fix: retrieve a wide set of candidates cheaply, then re-score them with something smarter before you build the prompt. This tutorial adds an LLM reranker, reciprocal rank fusion, context ordering and a groundedness check to a hybrid-search RAG bot in plain Node.js 22, with zero npm packages and a local model (Ollama with qwen2.5:3b and nomic-embed-text). Every step is measured on a small evaluation set, including the step where the first reranker made things worse.

⚡ TL;DR – RAG bots go wrong in four ways: wrong chunk, right chunk ranked low, misread context, or guessing when nothing answers. – Build a golden set first. In the video: 26 questions (22 answerable, 4 not), graded as correct, wrong, or “I don’t know”. – Baseline hybrid search scored 22/26 correct with 2 wrong. The first reranker dropped it to 17/26. Reranker v2 plus fusion got 22/26 with 0 wrong. – Handle stale pages in code (metadata), not in the prompt. – A groundedness check trades a few correct answers for fewer wrong ones. On a small set, run it twice; the results vary.

Before you start

• Node.js 22+ (built-in fetch).
• Ollama with two models: ollama pull qwen2.5:3b and ollama pull nomic-embed-text.
• A working retrieval pipeline: load, chunk, embed, hybrid search (BM25 + vectors). The RAG chatbot tutorial on this channel builds exactly that, and this one starts from it.
• A small docs corpus the model can’t already know. The video uses a made-up notes app help center, so correct answers must come from retrieval.

Why a RAG bot gives wrong answers

1. Retrieval finds the wrong chunk: something old, similar-sounding, or off-topic.
2. The right chunk is found but buried: ranked fifth, or sitting in the middle of a long context.
3. The model misreads the context.
4. Nothing answers the question and the model guesses anyway.

Reranking targets the first two. A groundedness check targets the last two. Keep this list in mind, because each fix only helps its own failure.

Step 1: Build the evaluation set before changing anything

Without measurement, every “fix” is a feeling. Write questions with expected answers and include some that the docs can’t answer:

// eval.json
[
  { "q": "How much is the Pro plan per month?", "expect": "$10" },
  { "q": "How many days is the refund window?", "expect": "14" },
  { "q": "Does Nimbus support Linux desktop?", "expect": null }  // unanswerable
]

Grade each answer as correct (contains the expected string), wrong (answered, but not the expected string), or idk (the bot declined). For unanswerable questions, “idk” is correct. Also log hit@1 and hit@3: was the right document the first result, or in the top three?

for (const item of evalSet) {
  const { answer, sources } = await ask(item.q);
  const declined = /i don't know/i.test(answer);
  let grade;
  if (item.expect === null) grade = declined ? 'correct' : 'wrong';
  else if (declined) grade = 'idk';
  else grade = answer.includes(item.expect) ? 'correct' : 'wrong';
  tally[grade]++;
}

String matching is crude, and the video says so. It’s still far better than eyeballing.

Step 2: Reproduce the failure

Add the stale changelog page, re-run ingest (load, chunk, embed; in the video 10 docs became 25 chunks of 768-dimension vectors), and run the eval. Hybrid search alone got 22/26 correct, with 2 confidently wrong answers, both quoting the archived prices. Those two wrong answers matter more than the score: a bot that says “I don’t know” is annoying, but a bot that quotes the wrong refund policy is a support ticket.

Step 3: Add an LLM reranker that returns JSON scores

Retrieve more candidates than you’ll use (say 8), then ask the model to score each one from 0 to 10 for how well it answers the question. Ask for JSON and enable Ollama’s JSON mode so the output parses:

async function rerank(question, chunks) {
  const scored = [];
  for (const c of chunks) {
    const out = await chat([
      { role: 'system', content: 'Score how well the passage answers the question. Reply as JSON: {"score": 0-10}.' },
      { role: 'user', content: `Question: ${question}\n\nPassage:\n${c.text}` },
    ], { json: true });                  // passes format: "json" to Ollama
    const score = Number(JSON.parse(out).score);
    scored.push({ ...c, rerank: Number.isFinite(score) ? score : 0 });
  }
  return scored.sort((a, b) => b.rerank - a.rerank);
}

Score one passage per call. Asking a 3B model to rank eight passages in one prompt produces inconsistent output.

Step 4: Understand why the first reranker made things worse

The first version dropped the score to 17/26. Replacing a decent ranking with a small model’s opinion threw away the signal BM25 and vectors already had. The model gave several chunks similar scores, ties broke badly, and the stale page, which really does talk about prices, scored high.

The lesson: a reranker is another noisy signal, not an oracle. Combine it with what you already have.

Step 5: Fuse rankings with reciprocal rank fusion

RRF combines rankings using only positions, so you don’t need to normalize scores from different systems:

function rrf(rankings, k = 60) {
  const scores = new Map();
  for (const list of rankings) {
    list.forEach((item, i) => {
      scores.set(item.id, (scores.get(item.id) ?? 0) + 1 / (k + i + 1));
    });
  }
  return [...scores.entries()].sort((a, b) => b[1] - a[1]).map(([id]) => id);
}

const fused = rrf([hybridResults, rerankedResults]);

A chunk that both the hybrid search and the reranker like rises to the top. A chunk only one of them likes drops back. With reranker v2 (one passage per call, a clearer scoring rubric) plus fusion, the eval went to 22/26 correct and 0 wrong.

Step 6: Handle stale pages in code, not the prompt

“Ignore outdated pages” in a system prompt is a request a small model will sometimes ignore. Put the fact in metadata and enforce it:

// at ingest: doc.meta = { title, url, updated: '2024-12-01', archived: true }
const usable = candidates.filter(c => !c.meta.archived);

If archived pages must stay searchable for history questions, apply a heavy penalty instead of filtering, and label them in the context (“ARCHIVED 2024:”) so citations make it obvious.

Step 7: Order the context against “lost in the middle”

Models pay most attention to the beginning and end of a long context. Put the best chunk first, the second-best last, and the weaker ones in between:

function orderForContext(chunks) {   // chunks sorted best-first
  if (chunks.length < 3) return chunks;
  return [chunks[0], ...chunks.slice(2), chunks[1]];
}

With only three or four chunks this matters less, but it costs nothing.

Step 8: Add a groundedness check

Check the draft answer before showing it. Do the cheap, deterministic checks in code first, then ask the model for a PASS/FAIL verdict:

function codeChecks(answer, context) {
  const nums = answer.match(/\$?\d+(\.\d+)?/g) ?? [];
  const missing = nums.filter(n => !context.includes(n));
  const cited = /\[\d+\]/.test(answer);
  return { ok: missing.length === 0 && cited, missing };
}
// if codeChecks passes, ask: "Is every claim supported by the context? Reply PASS or FAIL."
// on FAIL, return "I don't know" plus the closest sources.

Numbers that don’t appear anywhere in the retrieved text are the clearest sign of a guess. In the video the groundedness check moved the score to 21/26 with 0 wrong: one correct answer became “I don’t know”.

Step 9: Run it twice

With everything on, the same code scored 19/26 on one run and 21/26 on the next. Small models are not deterministic, even at low temperature, and 26 questions is a small set. Report ranges, not single numbers, and grow the eval set as real user questions come in.

Troubleshooting

• JSON.parse fails on reranker output: you forgot format: "json", or the model wrapped the JSON in prose. Fall back to a regex for the first number and log the raw text.
• Reranking is slow: it’s one model call per candidate. Rerank 8, not 50, or use a dedicated cross-encoder reranker model.
• Scores all come back 7 or 8: tighten the rubric (“10 = directly states the answer, 5 = related topic, 0 = unrelated”) and include one short example.
• Too many “I don’t know” answers: the groundedness check is too strict. Check the number regex against currency formats like $8/month.

Production checklist

• Golden set in the repo, run in CI on any retrieval change.
• Track wrong and “I don’t know” separately; don’t just track accuracy.
• Stale or archived content flagged at ingest.
• Reranker output validated, with a fallback to the fused hybrid ranking.
• Citations required, and checked in code.

FAQ

Should I use a dedicated reranker model instead of the chat model?

Usually, yes, in production. Cross-encoder rerankers are faster and more consistent than prompting a chat model. The chat-model version here shows the idea with zero extra dependencies.

Is 26 questions enough?

It’s enough to catch regressions like the 17/26 drop. It’s not enough to claim small improvements. Add every real failed question to the set.

Why accept fewer correct answers?

Because a wrong answer with a citation looks trustworthy. For support and policy questions, “I don’t know, here are the closest pages” is the safer failure.

Official documentation

• Ollama structured outputs / JSON mode
• Ollama embeddings
• Reciprocal Rank Fusion (Cormack et al.)

Key takeaways

• Measure first. A golden set turned “this feels better” into 22, 17, 22 and 21 out of 26.
• A reranker alone can hurt. Fuse it with the existing ranking.
• Enforce freshness with metadata, not prompts.
• Check numbers and citations in code before asking an LLM judge.
• Optimize for zero wrong answers, and accept more “I don’t know” answers to get there.