Incident ChatOps AI: Draft Change Tickets Without Auto-Remediating

ChatOps AIs that “just fix prod” are how you get a second outage. The useful product is the opposite: from a Slack alert, draft a change ticket and runbook steps grounded in CloudWatch/X-Ray citations—then wait for a human to approve any mutating tool.

⚡ TL;DR: Subscribe to alert events; gather metrics/logs with read-only IAM; draft ticket + dry-run commands; require explicit emoji/button approval before any write. Log the full agent transcript. Pair with AI On-Call Copilots and LLM Incident Runbooks.

Threat model: what auto-remediation actually breaks

Auto-remediators look clever in demos and catastrophic under partial failure:

• Restarting pods that were failing closed on a bad config → flap loop
• Scaling write capacity because of a thundering herd that needed shed load, not more RCU
• Rolling back a deploy that fixed a different symptom while the real bug was a dependency outage

The agent’s job is decision support, not actuator authority. Treat every mutating tool as a change ticket step that needs a human signature.

Read-only gathering

// chatops/draft.ts
export async function draftFromAlarm(alarm: CloudWatchAlarmEvent) {
  const signals = await Promise.all([
    logsInsights(alarm.metricQuery),
    xrayTraceSummaries(alarm.service),
    recentDeploys(alarm.service),
  ]);
  const draft = await bedrock.converse({
    system: "Propose remediation; NEVER claim you executed anything. Cite signal IDs.",
    messages: [{ role: "user", content: JSON.stringify({ alarm, signals }) }],
  });
  return {
    ticket: {
      title: `[sev] ${alarm.alarmName}`,
      body: draft.markdown,
      citations: signals.citationIds,
    },
    proposedCommands: draft.commands.map(asDryRun),
    mutate: false,
  };
}

function asDryRun(cmd: string) {
  if (/\b(apply|delete|rm|put-item|update-function-code)\b/i.test(cmd)) {
    return `# DRY-RUN ONLY\n# ${cmd}`;
  }
  return cmd;
}

✅ Citations mandatory; mutate flag false by default.
❌ Agent with kubectl delete pod bound to a Slack mention.

IAM split: gather vs elevate

{
  "Version": "2012-10-17",
  "Statement": [{
    "Sid": "ReaderOnly",
    "Effect": "Allow",
    "Action": [
      "cloudwatch:GetMetricData",
      "logs:StartQuery",
      "logs:GetQueryResults",
      "xray:GetTraceSummaries",
      "ecs:Describe*"
    ],
    "Resource": "*"
  }]
}

Mint elevation only after Slack approval; put the approver user ID and ticket ID into the STS session tags for forensics.

Approval UX

// Slack interactive button
export async function onApprove(action: SlackAction) {
  assertOnCall(action.user);
  assertChangeTicketOpen(action.ticketId);
  // Dual control for prod-touching tools
  await requestSecondApprover(action);
  await enqueueMutatingTool(action.commandId); // still audited
}

Human-in-the-loop patterns: keep approvals explicit as in AI On-Call Copilots. Grounding in Live Tail: LLM Incident Runbooks.

What the ticket must contain

## Proposed change
- Service: checkout-api
- Hypothesis: elevated DynamoDB throttles after deploy abc123
- Evidence: Logs Insights query Q-441, X-Ray trace 1-...
- Steps (dry-run):
  1. aws dynamodb describe-table ...
  2. # scale write capacity (APPROVAL REQUIRED)
- Rollback: redeploy previous alias weight 100%
- Blast radius: checkout write path only; reads unaffected

Reject drafts that lack citations or invent metric names. A verifier step can re-run the Logs Insights query ID and fail the draft if the query 404s.

Transcript retention and postmortems

Store every agent turn (prompt, tool args, tool results, Slack actions) under s3://incidents/{ticketId}/chatops/. Redact secrets with a allowlist of field names. Link the transcript from the postmortem template so “what did the bot suggest?” is one click — see Deterministic Replay.

Game-day drill script

1. Inject a synthetic alarm with known-good evidence
2. Confirm bot posts a draft ticket within 2 minutes
3. Attempt to trigger mutate without approval — must fail closed
4. Approve with on-call + second approver; confirm elevate role mint
5. Verify transcript lands and elevated session expires ≤15 minutes

Alert ingress without write paths

Subscribe the drafter to EventBridge/CloudWatch alarm notifications (or a SQS buffer), never to a webhook that can invoke shell. Parse the alarm ARN, map to owning service via a CMDB tag, and refuse to draft if the service lacks a runbook pointer. This keeps unknown alarms from becoming hallucinated remediations.

export async function ingress(evt: AlarmEvent) {
  const svc = await cmdb.lookup(evt.alarmArn);
  if (!svc?.runbookUrl) {
    await slack.post({ text: `No runbook for ${evt.alarmName}; human triage required.` });
    return;
  }
  return draftFromAlarm(evt);
}

Prefer a 30–60s debounce so flapping alarms do not spawn duplicate tickets. Deduplicate on alarmArn + stateTransitionTime window.

Closing checklist

• [ ] ChatOps IAM is read-only until approval mints a short-lived elevate role
• [ ] Every draft cites metrics/logs/deploy IDs
• [ ] Slack buttons record user + timestamp + ticket ID
• [ ] Mutating tools dual-controlled; transcript retained
• [ ] No auto-remediation webhooks from alert → shell
• [ ] Sev-1 runbooks tested in game days with the bot in draft-only mode
• [ ] Dry-run prefixer strips apply/delete verbs before posting
• [ ] STS session tags carry tenant/service/approver for audit

Related reading

• AI On-Call Copilots: Suggest Runbooks Without Mutating Production
• LLM Incident Runbooks: Ground On-Call Answers in CloudWatch Signals
• Deterministic Replay: Agent Sessions You Can Debug in Postmortems
• Secure AI Sandboxes: Ephemeral ECS Tasks for Agent Tool Execution

Last updated on September 11, 2026