Post-Incident AI Summaries: Ground Drafts in Timeline Artifacts Humans Edit

Blank-page postmortems delay learning; ungrounded LLM writeups invent confident fiction. The workable pattern is mechanical: assemble a timeline from PagerDuty, CloudWatch, X-Ray, and deploy markers, ask the model to draft only from those artifacts, then require a human editor before anything is published.

⚡ TL;DR: Export incident timeline JSON (alerts, deploys, graph annotations, chat excerpts). Prompt with “cite artifact IDs or stay silent.” Humans edit impact, root cause, and action items. Never auto-publish. Pair with LLM Incident Runbooks, Deterministic Replay, and Incident ChatOps AI.

Pipeline

PagerDuty + CloudWatch alarm history
  + CodeDeploy / ECS deployment events
  + X-Ray / OTel trace highlights
  + Slack #incident pinned messages (sanitized)
        ↓
  timeline.json (immutable artifact)
        ↓
  Bedrock draft (cite-or-silent)
        ↓
  Human edit + approve
        ↓
  Publish to wiki / Blameless

Timeline artifact schema

{
  "incident_id": "inc-2026-09-11-checkout",
  "events": [
    {"t": "2026-09-11T05:12:00Z", "type": "alarm", "id": "cw:PayErrorRate", "summary": "Pay 5xx > 2%"},
    {"t": "2026-09-11T05:10:00Z", "type": "deploy", "id": "ecs:checkout:17", "summary": "checkout task def 17"},
    {"t": "2026-09-11T05:18:00Z", "type": "mitigation", "id": "pd:ack", "summary": "rolled back to td 16"}
  ],
  "metrics": {"impact_orders": 842, "error_budget_burn": "3.2x/1h"},
  "forbidden": ["speculate beyond events", "name individuals beyond roles"]
}

const system = `You draft postmortem sections from timeline.json only.
Every factual claim must cite an event id. If unknown, write "UNKNOWN".
Do not invent root causes. Output markdown sections: Summary, Timeline, Impact, What went well, What went poorly, Action items (owners TBD).`;

const draft = await bedrock.converse({
  system: [{ text: system }],
  messages: [{ role: "user", content: [{ text: JSON.stringify(timeline) }] }],
});
// store draft + prompt hash + model id for audit (deterministic replay)

Human editing gates

# ✅ Publish path
# draft.md + timeline.json + editor sign-off in PR
# ❌ Model posts directly to the company wiki

Ground the same way you ground on-call assistants (LLM Incident Runbooks) and keep agent sessions replayable (Deterministic Replay). ChatOps may draft tickets but never auto-remediate (Incident ChatOps AI).

Closing checklist

• [ ] Timeline export is automated and immutable per incident
• [ ] Prompts enforce cite-or-silent; unknowns marked UNKNOWN
• [ ] Drafts store model id, prompt hash, and source artifact digests
• [ ] Human editor required before wiki publish
• [ ] Action items need real owners—AI suggestions are not tickets
• [ ] PII/secrets scrubbed from Slack excerpts before model context

Related reading

• LLM Incident Runbooks: Ground On-Call Answers in CloudWatch Signals
• Deterministic Replay: Agent Sessions You Can Debug in Postmortems
• Incident ChatOps AI: Draft Change Tickets Without Auto-Remediating
• AI On-Call Copilots: Suggest Runbooks Without Mutating Production

Last updated on September 11, 2026