Day 65: PII Redaction Before Embeddings

If you embed raw Jira and wiki text, you index people’s emails, phone numbers, and health notes into the vector DB forever. PII redaction belongs before chunking, not as a best-effort UI toggle.

⚡ TL;DR: Run deterministic + NER redaction in the ingest path. Store mapping tables separately with strict ACL. Re-embed on policy change.

Ingest pipeline

raw doc → PII redact → chunk → embed → index
         ↘ pii_map (ACL’d)

# ingest/pii.py
import re

EMAIL = re.compile(r"[A-Za-z0-9._%+-]+@[A-Za-z0-9.-]+\.[A-Za-z]{2,}")
PHONE = re.compile(r"\+?\d[\d\-\s]{8,}\d")

def redact_pii(text: str) -> tuple[str, dict]:
    mapping = {}
    def sub_email(m):
        token = f"EMAIL_{len(mapping)}"
        mapping[token] = m.group(0)
        return token
    out = EMAIL.sub(sub_email, text)
    out = PHONE.sub(lambda m: f"PHONE_{len(mapping)}", out)
    return out, mapping

Don’t embed the map

Keep pii_map in encrypted storage; retrieval for humans that need plaintext goes through an authorized reveal API — never through the model context by default.

Closing checklist

• [ ] Redact before embed
• [ ] Version redaction policy
• [ ] ACL the mapping table
• [ ] Re-index on policy tightening
• [ ] Test with synthetic PII canaries

Series navigation

Day 64: Secret-Aware Context Filters · Day 66: Audit Trails Humans Can Replay

Last updated September 11, 2026