LLM Incident Runbooks: Ground On-Call Answers in CloudWatch Signals

Generic runbook chatbots hallucinate remediations under sev-1 pressure. The unfair advantage is grounding every suggested action in live CloudWatch signals — Live Tail snippets, Logs Insights aggregates, X-Ray traces, and alarm state — so the assistant cites evidence before it proposes a rollback, a scale-out, or a feature-flag flip.

⚡ TL;DR: Build a retrieval + tool loop that pulls alarm context, runs pre-approved Logs Insights queries, fetches recent X-Ray traces, and only then asks the model to draft a remediation. Require citations (logGroup, queryId, traceId) on every claim. Cap blast radius with allowlisted actions. Pair with Lambda Powertools structured logs and timeouts/retries/DLQs so signals exist when you need them.

Bind the assistant to alarm and service context

On page, inject the firing alarm ARN, dimensions, and service map — never a free-form “what’s wrong?” with no scope.

// incident-bot/context.ts
export type IncidentContext = {
  alarmArn: string;
  service: string;          // e.g. checkout-api
  env: "prod" | "staging";
  region: string;
  startedAt: string;        // ISO
  severity: "SEV1" | "SEV2" | "SEV3";
};

export function systemPrompt(ctx: IncidentContext): string {
  return [
    "You are an on-call assistant. Cite CloudWatch/X-Ray evidence for every claim.",
    `Service=${ctx.service} env=${ctx.env} region=${ctx.region} sev=${ctx.severity}`,
    `Alarm=${ctx.alarmArn} started=${ctx.startedAt}`,
    "Never invent metric names. Prefer tool results over memory.",
    "Remediations must map to allowlisted runbook IDs.",
  ].join("\n");
}

✅ Scope the model to one service + one alarm window.
❌ Dump the entire company wiki and hope it finds the right playbook.

Pre-approve Logs Insights queries as tools

Do not let the model invent arbitrary Insights queries against * log groups. Ship parameterized, reviewed queries.

# tools/logs_insights.py
import time, boto3

logs = boto3.client("logs")

APPROVED = {
  "error_rate_by_path": """
fields @timestamp, path, status
| filter status >= 500 and service = '{service}'
| stats count() as errors by path
| sort errors desc
| limit 20
""",
  "p99_latency": """
fields @timestamp, latencyMs, path
| filter service = '{service}'
| stats pct(latencyMs, 99) as p99 by path
| sort p99 desc
| limit 20
""",
}

def run_approved(query_id: str, service: str, log_group: str, start: int, end: int) -> dict:
    if query_id not in APPROVED:
        raise ValueError("query_not_allowlisted")
    q = APPROVED[query_id].format(service=service)
    # ❌ Never: interpolate raw model text into the query string
    start_q = logs.start_query(
        logGroupName=log_group,
        startTime=start,
        endTime=end,
        queryString=q,
    )
    qid = start_q["queryId"]
    while True:
        r = logs.get_query_results(queryId=qid)
        if r["status"] in ("Complete", "Failed", "Cancelled"):
            return {"queryId": qid, "status": r["status"], "results": r.get("results", [])}
        time.sleep(0.4)

Wire Live Tail only for short windows (illustrative: last 2–5 minutes) and stream truncated lines with secrets redacted — same discipline as Powertools structured logs.

Pull X-Ray traces before blaming “the database”

// tools/xray.ts
import { XRayClient, GetTraceSummariesCommand, BatchGetTracesCommand } from "@aws-sdk/client-xray";

const xray = new XRayClient({});

export async function slowTraces(service: string, start: Date, end: Date) {
  const summaries = await xray.send(new GetTraceSummariesCommand({
    StartTime: start,
    EndTime: end,
    FilterExpression: `service("${service}") AND responseTime > 2`,
  }));
  const ids = (summaries.TraceSummaries ?? []).slice(0, 5).map(t => t.Id!).filter(Boolean);
  if (!ids.length) return { traces: [], note: "no_slow_traces" };
  const batch = await xray.send(new BatchGetTracesCommand({ TraceIds: ids }));
  // ✅ Return truncated segment names + durations, not full payloads with PII
  return {
    traces: (batch.Traces ?? []).map(t => ({
      id: t.Id,
      duration: t.Duration,
      segments: (t.Segments ?? []).slice(0, 12).map(s => s.Document?.slice(0, 400)),
    })),
  };
}

Require the model’s answer format to include evidence: [{type, id, snippet}]. Reject replies without evidence in a validator — treat uncited remediations as failed tool use.

Allowlist remediations; never open-ended shell

# runbooks/allowlist.yaml
actions:
  - id: rollback_ecs_service
    requires: ["evidence.deploy_marker", "sev<=SEV2"]
    params: ["service", "previousTaskDef"]
  - id: scale_out_asg
    requires: ["evidence.cpu_or_latency"]
    params: ["asgName", "desiredCapacity"]
    maxDesired: 20
  - id: flip_feature_flag
    requires: ["evidence.error_burst"]
    params: ["flagKey", "value"]
# ❌ no: arbitrary aws cli / kubectl from the model

function assertAction(action: { id: string; params: Record<string, unknown> }, allow: Set<string>) {
  if (!allow.has(action.id)) throw new Error(`deny:${action.id}`);
  // ✅ Human approval gate for SEV1 destructive actions
}

For agents that can mutate infra, sandbox execution like Lambda tool sandboxes and Bedrock Agents guardrails.

Observability of the incident bot itself

Emit: tool latency, citation rate, human override rate, wrong-query denials. Illustrative SLO: ≥95% of SEV1 answers include ≥1 Logs Insights or X-Ray citation; median tool round-trip under ~3s for Insights on warm queries.

Closing checklist

✅ Dos
– ✅ Inject alarm + service + time window into the system prompt
– ✅ Allowlist Logs Insights query templates; parameterize service only
– ✅ Cite queryId / traceId on every remediation claim
– ✅ Gate destructive actions behind human approval + allowlist
– ✅ Redact secrets from Live Tail before model context

❌ Don’ts
– ❌ Don’t let the model invent Insights queries against *
– ❌ Don’t accept remediations without evidence blocks
– ❌ Don’t grant the bot * IAM for “faster incident response”
– ❌ Don’t stream raw request bodies with PII into the LLM
– ❌ Don’t skip structured logging — you cannot ground what you never emitted

Related reading

• Lambda Powertools for Node: Structured Logs That Survive On-Call
• Lambda Timeouts, Retries, and DLQs: Idempotent Failure Handling
• LLM Coding Agents on AWS: Safe Tool Sandboxes with Lambda
• Amazon Bedrock Agents: Tool Use, Memory, and Production Guardrails

Last updated on September 11, 2026