Embedding Drift Alerts: Catch Provider Vector Changes Before RAG Dies

Providers ship new embedding versions; your index stays on yesterday’s space. Cosine neighbors become nonsense, hit rate tanks, and every coding assistant starts citing the wrong module—quietly. Drift alerts on a frozen canary corpus catch the cliff before RAG dies in production.

⚡ TL;DR: Keep a pinned canary set of (query → expected chunk IDs). Re-embed canaries on a schedule and after model ID changes; alarm when recall@k or centroid shift exceeds thresholds; trigger partial/full reindex. Never mix embedding model versions in one index. Pair with Codebase Embeddings Refresh Pipelines and RAG Evaluation on AWS.

Canary corpus that actually fails loudly

# drift/canary.py
from dataclasses import dataclass
import numpy as np

@dataclass
class Canary:
    query: str
    expected_ids: list[str]  # gold chunk ids
    emb_model: str

def recall_at_k(neighbors: list[str], expected: list[str], k: int = 5) -> float:
    top = set(neighbors[:k])
    return len(top & set(expected)) / max(1, len(expected))

def centroid_shift(old: np.ndarray, new: np.ndarray) -> float:
    # mean L2 between paired canary embeddings
    return float(np.linalg.norm(old - new, axis=1).mean())

def assert_healthy(recalls: list[float], shift: float):
    if np.median(recalls) < 0.8:
        raise SystemExit(f"recall_collapse:{np.median(recalls):.3f}")
    if shift > 0.35:  # tune per model family
        raise SystemExit(f"embedding_space_shift:{shift:.3f}")

Store canaries in S3 versioned JSON; regenerate only with human review. Tie offline eval harnesses to RAG Evaluation hit-rate curves.

Detect provider upgrades

// Always pin model IDs in config — never "latest"
export const EMBEDDING = {
  modelId: "amazon.titan-embed-text-v2:0",
  dim: 1024,
  indexAlias: "code-rag-v2",
};

// On deploy / cron:
async function probeProvider(modelId: string) {
  const meta = await bedrock.getFoundationModel({ modelIdentifier: modelId });
  // Persist content hash / ARN; alarm on unexpected change
  await putMetric("EmbeddingModelFingerprint", hash(meta));
}

✅ Explicit model ID + dim + index alias.
❌ One OpenSearch index fed by two embedding model versions.

Mixed-version poison

The failure mode is subtle: overnight job re-embeds 5% of chunks on titan-v2 while the rest remain titan-v1. kNN returns incoherent neighborhoods. Guard writes:

def upsert(chunk_id: str, vector, model_id: str, index_model_id: str):
    if model_id != index_model_id:
        raise ValueError(f"model_mismatch:{model_id}!={index_model_id}")
    client.index(id=chunk_id, body={"vector": vector, "model_id": model_id})

Store model_id on every vector document; reject queries that omit a matching filter when dual indexes temporarily coexist during migration.

Reindex playbook

1. Alarm fires (recall or shift).
2. Spin dual-write to code-rag-v3 with new model.
3. Backfill changed paths first (see incremental jobs in embeddings refresh).
4. Shadow queries compare v2 vs v3 hit rate.
5. Flip alias; tombstone old index after soak.

# illustrative
aws opensearch update-alias --name code-rag --add code-rag-v3 --remove code-rag-v2

Closing checklist

• [ ] Canary corpus checked into git with owners
• [ ] Cron + post-deploy probe of embedding model fingerprint
• [ ] Alarms on recall@5 median and centroid shift
• [ ] Index alias strategy; no mixed-version vectors
• [ ] Dual-write + shadow before alias flip
• [ ] Budget for full reindex documented (hours + $)

Related reading

• Codebase Embeddings: Refresh Pipelines When Git SHAs Keep Moving
• RAG Evaluation on AWS: Hit Rate, Faithfulness, and Cost Curves
• Bedrock Knowledge Bases: Chunking Strategies That Fit Code RAG
• Hybrid Search for Code: BM25, Vectors, and Symbol Indexes Together

Last updated on September 11, 2026