RAG Evaluation on AWS: Hit Rate, Faithfulness, and Cost Curves

Shipping a new chunker or prompt because a demo “felt better” is how RAG quality regresses silently. Build an offline harness on S3 + Batch (or Step Functions) that measures retrieval hit rate, answer faithfulness, citation accuracy, and Bedrock $ / query before anything reaches prod aliases.

⚡ TL;DR: Freeze a golden question set with expected file/symbol citations; run retrieval + generation in Batch against candidate configs; score hit@k, faithfulness, citation precision; plot cost curves from token metrics; promote only if quality ≥ baseline and cost ≤ budget. See OpenSearch vs pgvector and prompt caching.

Golden set that matches how engineers ask

{
  "id": "q-042",
  "question": "Where do we enforce tenant isolation on order reads?",
  "expect_paths": ["packages/orders/src/getOrder.ts"],
  "expect_symbols": ["assertTenant"],
  "difficulty": "medium"
}

Keep 100–500 questions. Tag by area (auth, payments, infra). Refresh when major modules move — same problem as embedding freshness (companion topic in this batch).

Metrics that catch real failures

def hit_at_k(retrieved_paths: list[str], expect: list[str], k: int = 5) -> float:
    top = set(retrieved_paths[:k])
    return 1.0 if any(e in top for e in expect) else 0.0

def citation_precision(answer_cites: list[str], retrieved: list[str]) -> float:
    if not answer_cites:
        return 0.0
    ok = sum(1 for c in answer_cites if c in retrieved)
    return ok / len(answer_cites)

# Faithfulness: entailment model or Bedrock judge with rubric
FAITHFULNESS_RUBRIC = """
Score 1-5: does the answer only claim facts supported by the passages?
5 = fully supported; 1 = contradicts or invents APIs.
Return JSON {"score": int, "rationale": str}
"""

✅ Hit rate without faithfulness → confident wrong answers.
❌ BLEU on code answers — almost useless for API guidance.

Batch harness shape on AWS

# batch/evaluate_job.py
import json, boto3, os
s3 = boto3.client("s3")
brt = boto3.client("bedrock-runtime")

def main():
    bucket, key = os.environ["GOLDEN_S3"].split("/", 3)[2], os.environ["GOLDEN_S3"].split("/", 3)[3]
    # Actually parse s3://bucket/key properly in prod
    questions = json.loads(s3.get_object(Bucket=os.environ["BUCKET"], Key=os.environ["GOLDEN_KEY"])["Body"].read())
    rows = []
    for q in questions:
        retrieved = retrieve(q["question"], k=8)  # OpenSearch or pgvector
        answer, usage = generate(q["question"], retrieved)
        rows.append({
            "id": q["id"],
            "hit@5": hit_at_k([r["path"] for r in retrieved], q["expect_paths"], 5),
            "cite_prec": citation_precision(answer["cites"], [r["path"] for r in retrieved]),
            "faithfulness": judge(answer["text"], retrieved),
            "input_tokens": usage["input"],
            "output_tokens": usage["output"],
            "cost_usd": estimate_cost(usage),
        })
    s3.put_object(Bucket=os.environ["BUCKET"], Key=os.environ["OUT_KEY"], Body=json.dumps(rows).encode())

Store each run under s3://rag-eval/{git_sha}/{config_id}/results.json. Compare against baseline.json in CI for prompt PRs.

Cost curves before you “just use the biggest model”

# Illustrative planning numbers — label as such
# Haiku-class retrieve+answer: ~$0.002–0.008 / query
# Sonnet-class: ~$0.01–0.04 / query
# With prompt caching on large system+tool schemas: 30–70% input discount (see companion post)

Plot mean faithfulness vs mean $ / query across configs. Promote the Pareto-efficient point that clears your quality floor (illustrative: hit@5 ≥ 0.78, faithfulness ≥ 4.0/5, cite_prec ≥ 0.85).

Compare index backends with the same golden set — OpenSearch vs Aurora pgvector.

Closing checklist

✅ Dos
– ✅ Freeze golden questions with expected paths/symbols
– ✅ Measure hit@k, faithfulness, citation precision, $ / query
– ✅ Run offline on Batch/S3 before alias promotion
– ✅ Version results by git SHA + config id
– ✅ Enforce quality floor + cost ceiling in CI

❌ Don’ts
– ❌ Don’t promote prompts from vibe checks alone
– ❌ Don’t judge faithfulness without retrieved passages in the judge prompt
– ❌ Don’t ignore token usage metrics from Bedrock
– ❌ Don’t mix train questions into online eval without leakage controls
– ❌ Don’t delete prior eval runs — you need regressions later

Related reading

• RAG on AWS: OpenSearch vs Aurora pgvector for Codebase Chat
• Bedrock Prompt Caching and Batch Inference: Cut Latency and Cost
• Amazon Bedrock Agents: Tool Use, Memory, and Production Guardrails
• AI Code Review Bots: IAM, Secrets, and Least-Privilege Pipelines

Last updated on September 11, 2026