Day 3: Embeddings That Survive Domain Jargon

If your corpus is Terraform, PagerDuty runbooks, and service names like payments-ledger-v3, an embedding model tuned for encyclopedic English will retrieve the wrong neighbors with high confidence. Day 3 is about domain survival: selection, versioning, and evaluation — not crowning the public leaderboard winner.

⚡ TL;DR: Build a labeled query→passage set from real tickets. Score Recall@k and nDCG. Version embedding model IDs in index metadata. Re-embed on model change; never mix vector spaces without a migration plan.

What “survives jargon” means

A domain-competent embedding should pull these near each other when operators mean the same incident:

• ASG ↔ Auto Scaling group
• 403 from ALB ↔ listener rule denied
• ExpiredToken from boto3 ↔ refresh SSO / assume-role failure

And it should keep dangerous lookalikes apart:

• s3:DeleteBucket policy guidance vs “delete unused ECR images” cleanup runbook
• Identically worded staging vs prod incident notes

Generic web embeddings often collapse the lookalikes and miss the synonyms. That is a product bug, not a minor recall dip.

Choose with an evaluation set, not vibes

Mine 50–200 real queries from Slack, tickets, and search logs. For each, label one or more relevant chunk IDs. Then compare models on the same corpus snapshot.

# ✅ Offline eval you can put in CI
from dataclasses import dataclass

@dataclass
class Case:
    query: str
    relevant_ids: set[str]

def recall_at_k(ranked_ids: list[str], relevant: set[str], k: int) -> float:
    return len(relevant.intersection(ranked_ids[:k])) / max(1, len(relevant))

def eval_model(embed_fn, corpus, cases, k=10) -> float:
    vectors = {doc.id: embed_fn(doc.text) for doc in corpus}
    scores = []
    for c in cases:
        ranked = top_k_cosine(embed_fn(c.query), vectors, k)
        scores.append(recall_at_k(ranked, c.relevant_ids, k))
    return sum(scores) / len(scores)

❌ Shipping Titan, Cohere, or OpenAI embeddings because a blog ranked them #1 — without a domain query set. Leaderboards are not your on-call language.

Track failure cases in a spreadsheet: “query about blue/green ECS that retrieved Kubernetes Ingress docs.” Those clusters drive chunking and metadata fixes, not just model swaps.

Version embeddings like schema migrations

Every vector row needs provenance:

{
  "id": "runbook:rds-failover:v3",
  "embedding_model": "amazon.titan-embed-text-v2:0",
  "embedding_dim": 1024,
  "embedded_at": "2026-09-11T00:00:00Z",
  "content_sha": "…"
}

Hard rules:

• Never query with model A against an index built by model B.
• Blue/green re-embed: dual-write or rebuild, then atomic alias swap.
• Chunk text change ⇒ re-embed that chunk; delete orphans.
• Dimension changes require a new index or field — do not cast blindly.

Code and ops specific tactics

• Prefer models with strong code or asymmetric retrieval training when queries are short and documents are long.
• Lightly normalize identifiers (PaymentsLedger → payments ledger) for the dense channel, but keep exact symbols in BM25/keyword search (Day 21).
• Embed title + section headers + body; title-only vectors are too thin for jargon-heavy ops docs.
• For multi-tenant KBs, embed after ACL metadata is attached so filters cannot be “forgotten” at query time (Day 23).

When two models tie on Recall@10, pick the one with clearer versioning, regional availability on Bedrock, and predictable cost — operational excellence beats a 1% nDCG flex.

Closing checklist

• [ ] Collect ≥50 real queries with labeled passages from your domain
• [ ] Compare ≥2 embedding models on Recall@10 and nDCG
• [ ] Persist embedding_model, dim, and content_sha on every vector
• [ ] Document re-embed migration before any model swap
• [ ] Keep BM25/symbol search as a peer channel for exact jargon
• [ ] Review top confusion pairs monthly with on-call engineers

Worked example: jargon eval in one afternoon

Collect 40 queries from the last month of #oncall. For each, paste the runbook section that actually resolved the issue as the labeled passage. Embed with Model A and Model B. Chart Recall@10. You will usually see a clear winner on acronyms and a different winner on long prose — pick for your dominant query shape, or go hybrid with BM25.

report = []
for model_name, embed in CANDIDATES.items():
    report.append((model_name, eval_model(embed, corpus, cases, k=10)))
print(sorted(report, key=lambda x: -x[1]))

Store the losing model’s confusion pairs; they become chunking and metadata tasks, not just “try another vendor.”

Failure modes to watch

• Mixed vector spaces after a quiet model upgrade.
• Embedding titles only while queries mention error strings in bodies.
• No content hash → stale vectors after doc edits.
• Over-normalization that collapses prod vs staging tokens.

Field notes from production

When legal asks for deletion, you must find vectors by content_sha and source URI, not by fuzzy text search. Build a tombstone pipeline: delete source → delete chunks → delete vectors → audit row. Embedding model deprecations on Bedrock need the same seriousness as Postgres major upgrades — calendar them.

Implementation sketch

# Implementation sketch: refuse mixed spaces at query time
def query(qvec, meta_index, model_id):
    if meta_index.embedding_model != model_id:
        raise RuntimeError("embedding_model_mismatch")
    return ann_search(qvec, meta_index)

Operator addendum

Re-embed jobs should be resumable and checkpointed by content_sha. Partial failures that leave half an index on the new model are worse than delaying the cutover. Practice the alias swap in staging weekly until it is boring.

Series navigation

← Day 2 · Day 4 →

Last updated September 11, 2026