Day 18: Logging Prompts Without Leaking Secrets

You cannot debug agents without logs, and you cannot keep jobs if logs contain raw secrets and customer PII forever. Day 18 builds a logging pipeline with redaction, hashed identifiers, purpose-limited retention, and access controls legal and security can defend.

⚡ TL;DR: Never log raw secrets, session tokens, or unrestricted PII. Redact before write. Hash user IDs for joins. Separate debug payloads with short TTL from metrics with long TTL. Gate engineer access via JIT roles.

What to log vs not

# ✅ Redact before the logger
SECRET = re.compile(r"(AKIA[0-9A-Z]{16}|Bearer\s+[A-Za-z0-9._-]+|ghp_[A-Za-z0-9]+)")

def safe(text: str) -> str:
    return SECRET.sub("[REDACTED]", text)

Extend with company-specific patterns (connection strings, ID numbers). Prefer structured fields over giant strings.

Retention and access

• Debug prompt vault: 7–14 days, encrypted, JIT access, audited reads.
• Metrics/aggregates: 90+ days OK without raw text.
• Hash user_id with a keyed HMAC for correlation across services.

When an incident needs raw prompts, copy into a ticket-controlled secure store — do not widen CloudWatch retention casually.

Closing checklist

• [ ] Redaction library on all prompt/tool log paths
• [ ] Split short-TTL debug vs long-TTL metrics
• [ ] HMAC user identifiers
• [ ] JIT access + audit for raw debug vault
• [ ] Legal-reviewed retention statement
• [ ] Tests that fail if secrets appear in sample logs

Worked example: CI secret-in-logs test

Fixture prompts contain fake AWS keys. Run the logging pipeline in unit tests; assert output sinks never contain AKIA. Fail the build if a new log line bypasses safe().

Failure modes to watch

• Debug=true in prod dumping full prompts to shared Slack.
• Vendor LLM logs enabled without DPA review.
• Forever retention “just in case.”
• Redaction only on user text, forgetting tool results with secrets.

Field notes from production

Include a data-flow diagram in the DPA packet: where prompts go (Bedrock region, logs, eval stores). Engineers invent new sinks (temporary S3 debug buckets) — scan IaC for log destinations quarterly. Redaction tests belong in the same CI job as unit tests.

Implementation sketch

# Implementation sketch: dual sinks
metrics.emit(counts)                 # long TTL
debug_vault.put(safe(prompt), ttl=7) # short TTL, JIT

Operator addendum

When using third-party tracing tools, verify whether they store prompt bodies by default. Many do — turn it off or mask before you paste API keys into a ‘quick trial.’

Vendor subprocessors

Keep a living list of where prompt text may flow: model API, log vendor, eval vendor, support tooling. Each onboarding requires security review and redaction config screenshots. Engineers spinning up “just a LangSmith project” can violate the list — detect with egress controls where feasible and with education always. Include this list in onboarding for AI feature teams.

Extended discussion

Return to the core angle for Day 18: Redaction, hashed user ids, and retention that legal can defend. That sentence is the acceptance lens for every design review this week. If a proposed change does not make this angle easier to measure or enforce, it is a distraction.

Write down three metrics you will look at after shipping Day 18 ideas, schedule a 45-minute readout, and archive the notes next to the eval artifacts. Architecture without a readout becomes slideshow archaeology.

Pair this day with the adjacent lessons in the series navigation below. Forward links exist so you can keep momentum; backward links exist so you can repair foundations when a later lab fails for boring earlier reasons.

Practically, allocate half a day to implement the smallest vertical slice, half a day to wire measurement, and refuse to polish UI until both are done. This ordering is how bootcamp projects stay honest under time pressure.

Revisit assumptions whenever the model ID, embedding ID, or index alias changes — treat those as breaking changes for Day 18 behaviors, with the same seriousness as a database migration. Canary first, then promote.

Revisit assumptions whenever the model ID, embedding ID, or index alias changes — treat those as breaking changes for Day 18 behaviors, with the same seriousness as a database migration. Canary first, then promote.

Revisit assumptions whenever the model ID, embedding ID, or index alias changes — treat those as breaking changes for Day 18 behaviors, with the same seriousness as a database migration. Canary first, then promote.

Revisit assumptions whenever the model ID, embedding ID, or index alias changes — treat those as breaking changes for Day 18 behaviors, with the same seriousness as a database migration. Canary first, then promote.

Revisit assumptions whenever the model ID, embedding ID, or index alias changes — treat those as breaking changes for Day 18 behaviors, with the same seriousness as a database migration. Canary first, then promote.

Revisit assumptions whenever the model ID, embedding ID, or index alias changes — treat those as breaking changes for Day 18 behaviors, with the same seriousness as a database migration. Canary first, then promote.

Series navigation

← Day 17 · Day 19 →

Last updated September 11, 2026