Day 90: Project: Incident Timeline Summarizer

Lab / Project: Incident Timeline Summarizer

Post-incident Slack walls are unreadable. Build a summarizer that grounds every claim in artifacts (pages, deploys, metric links, ticket ids) and refuses to post until a human edits.

⚡ TL;DR: Ingest PagerDuty/Opsgenie, deploy markers, and chat exports. Draft a timeline with citations. Human edits in a doc. Only then allow “post to #incidents.” No silent auto-post.

Pipeline

Artifacts → normalize → retrieve → draft timeline → human edit → optional post

# ✅ Citation-forced draft
def draft_timeline(artifacts: list[dict]) -> dict:
    evidence = index(artifacts)
    draft = llm_draft(evidence)
    for claim in draft["claims"]:
        if claim["cite_id"] not in evidence:
            claim["status"] = "ungrounded"
    draft["claims"] = [c for c in draft["claims"] if c["status"] != "ungrounded"]
    draft["ready_to_post"] = False
    return draft

Acceptance criteria

• [ ] Every bullet cites an artifact id
• [ ] UI blocks post while ready_to_post is false
• [ ] Redacts customer names by default
• [ ] Stores draft + final under incident id
• [ ] Eval: replay 5 past SEVs; faithfulness score ≥ threshold

❌ Auto-posting LLM drafts into customer-facing status pages.

Failure modes

Hallucinated timeline order when clock skew exists across chat and paging tools. Normalize all timestamps to UTC and sort with source priority rules. Auto-post integrations that ignore ready_to_post — remove the bot token’s chat:write until freeze.

Closing checklist

• [ ] Artifact connectors with least privilege
• [ ] Human edit gate
• [ ] Redaction layer
• [ ] Faithfulness eval harness
• [ ] Runbook for summarizer outage (write manually)

Series navigation

Day 89: On-Device and Edge Completions · Day 91: Architecture Review: Draw the Box Diagram First

Last updated September 11, 2026