AI audit trails: Audit Trails Humans Can Replay

Day 66: Audit Trails Humans Can Replay
12 views
This guide focuses on AI audit trails for production systems, with practical trade-offs for reliability, security, and cost. “The agent did something weird” is not a postmortem. You need deterministic session logs — prompts (redacted), tool calls, artifacts hashes, model IDs — that a human can replay without guessing.
⚡ TL;DR: Append-only event log per run_id. Store artifact hashes, not only chat. Replay tool stubs in a dry-run harness.

AI audit trails: production guidance

json
{
  "run_id": "run_7f3a",
  "seq": 12,
  "type": "tool_call",
  "tool": "apply_intent",
  "args_hash": "…",
  "result_hash": "…",
  "model": "anthropic.claude-…",
  "ts": "2026-09-11T15:01:02Z"
}
python
# audit/log.py
import json, hashlib
from pathlib import Path

def append_event(run_dir: Path, event: dict):
    event = dict(event)
    event["event_hash"] = hashlib.sha256(
        json.dumps(event, sort_keys=True).encode()
    ).hexdigest()
    with (run_dir / "events.jsonl").open("a") as f:
        f.write(json.dumps(event) + "\n")  # ✅ append-only

Replay

bash
python audit/replay.py --run artifacts/run_7f3a --dry-run
Replay should re-validate hashes and re-execute pure tools; impure tools stay stubbed with recorded results.

Closing checklist

  • [ ] JSONL append-only events
  • [ ] Hash args/results
  • [ ] Record model + prompt version
  • [ ] Redact secrets/PII in logs
  • [ ] Human replay dry-run tool

Series navigation

Day 65: PII Redaction Before Embeddings · Day 67: Supply Chain: Model and Image Provenance Last updated September 11, 2026 Related: Post-Incident AI Summaries: Ground Drafts in Timeline Artifacts Humans Edit; Day 76: p99 of Agents: Queueing, Not Just Model Latency; Day 77: Caching RAG Answers Safely.

Watch: 100 Days of AI on YouTube

Short videos from CheatCoders, one AI engineering topic per day.

Open the playlist on YouTube · CheatCoders on YouTube

Deep-dive PDF

Get the expanded guide for this post — extra diagrams-style checklists, failure modes, and a production walkthrough. Free when you subscribe to CheatCoders.

Already subscribed? or open the subscribe page.


Discover more from CheatCoders

Subscribe to get the latest posts sent to your email.

Comments

No comments yet. Why don’t you start the discussion?

Leave a comment

No account needed. Name and email are optional.