Deterministic Replay: Agent Sessions You Can Debug in Postmortems

When an agent ships a bad change, “the model felt like it” is not a root cause. You need a replayable session transcript: prompts, tool I/O, model IDs, decoding params, and git SHAs — enough to reconstruct what the agent saw and did during the incident window.

⚡ TL;DR: Emit an append-only session log (JSONL) with redacted prompts, tool spans, model version, seed/temperature, and repo SHA. Replay offline against recorded tool stubs. Never require live Bedrock for postmortems. Pair with OpenTelemetry for LLMs and LLM Incident Runbooks.

Session log schema

{
  "session_id": "ses_01J...",
  "ts": "2026-09-11T07:12:01.412Z",
  "type": "model_call",
  "model_id": "anthropic.claude-sonnet-4-20250514-v1:0",
  "params": {"temperature": 0, "top_p": 1},
  "repo_sha": "abc123",
  "prompt_hash": "sha256:...",
  "prompt_redacted_uri": "s3://agent-logs/.../prompt.json",
  "output_hash": "sha256:...",
  "tool_calls": [{"id": "call_1", "name": "apply_patch"}]
}

// replay/log.ts
export type Span =
  | { type: "model_call"; model_id: string; params: Record<string, unknown>; prompt_hash: string; output: string }
  | { type: "tool_call"; name: string; input: unknown; output: unknown; duration_ms: number }
  | { type: "git"; sha: string; branch: string };

export function append(sessionId: string, span: Span) {
  // ✅ Append-only; WORM bucket recommended
  return s3.put(`${sessionId}/${Date.now()}.json`, JSON.stringify(span));
}

Redact before persistence

SECRET = re.compile(r"(AKIA[0-9A-Z]{16}|Bearer\s+\S+|-----BEGIN [A-Z ]+PRIVATE KEY-----)")

def redact(text: str) -> str:
    return SECRET.sub("[REDACTED]", text)

Same hygiene as secret-aware context filters. Retention: 30–90 days with legal hold for sev-1 sessions.

Offline replay harness

async function replay(sessionId: string) {
  const spans = await loadSpans(sessionId);
  const toolStub = new Map<string, unknown>();
  for (const s of spans) {
    if (s.type === "tool_call") toolStub.set(key(s), s.output);
  }
  for (const s of spans) {
    if (s.type === "model_call") {
      // ✅ Compare recorded output hash; optionally re-call model only if investigating nondeterminism
      assertHash(s.output, s /* stored */);
    }
    if (s.type === "tool_call") {
      const recorded = toolStub.get(key(s));
      // Replay consumers read recorded outputs — no live AWS
      applyForDebug(s.name, recorded);
    }
  }
}

❌ Replaying by calling production tools again — that doubles the blast radius.

What postmortems should answer

Closing checklist

✅ Dos
– ✅ JSONL session logs with model + git SHA + tool I/O
– ✅ Redact secrets; encrypt at rest
– ✅ Replay from stubs, not live prod tools
– ✅ Link session_id from the PR and deploy
– ✅ Include Prompt Management / guardrail versions

❌ Don’ts
– ❌ Don’t store raw secrets “for debugging later”
– ❌ Don’t rely on chat UI history alone
– ❌ Don’t re-execute destructive tools during replay
– ❌ Don’t omit temperature/top_p when nonzero

Related reading

• OpenTelemetry for LLMs: Trace Prompt Latency Across Microservices
• LLM Incident Runbooks: Ground On-Call Answers in CloudWatch Signals
• Secret-Aware Context Filters
• Bedrock Prompt Management: Versioned System Prompts for Production

Last updated on September 11, 2026