AI On-Call Copilots: Suggest Runbooks Without Mutating Production

On-call copilots that can kubectl delete without a human are an outage amplifier. Ship read-only assistants that draft remediations with dry-run flags, cite CloudWatch/X-Ray evidence, and open change tickets — then wait for explicit engineer approval before any write. Suggestion ≠ execution.

⚡ TL;DR: Default tool allowlist is get/list/describe/logs only. Writes require dual-control approval. Every suggestion cites signals. Ground answers like LLM incident runbooks and gate shells like Claude Code hooks.

Read-only by construction

// oncall_tools.ts
const READ_ONLY = [
  /^kubectl (get|describe|logs|top) /,
  /^aws (logs|cloudwatch|xray|ecs describe|dynamodb describe)/,
  /^aws sts get-caller-identity$/,
];

const WRITEISH = [
  /kubectl (apply|delete|scale|rollout restart|exec)/,
  /aws (ecs update|autoscaling|s3 rm|dynamodb delete)/,
];

export function assertReadOnly(cmd: string) {
  if (WRITEISH.some((re) => re.test(cmd))) {
    throw new Error("write_requires_approval"); // ✅ hard stop
  }
  if (!READ_ONLY.some((re) => re.test(cmd))) {
    throw new Error("command_not_allowlisted");
  }
}

// ❌ Agent tool: run_shell(cmd) with no classifier

✅ Propose kubectl delete pod X --dry-run=server as text.
❌ Execute delete because the model “is confident.”

Ground every suggestion in signals

# suggest_remediation.py
def suggest(sev_context: dict) -> dict:
    evidence = {
        "logs": logs_insights(sev_context["query"]),
        "alarms": cloudwatch_alarms(sev_context["service"]),
        "trace": xray_trace(sev_context.get("traceId")),
    }
    draft = bedrock_converse(
        system="Propose remediations. Cite evidence IDs. Never claim you executed writes.",
        user=json.dumps(evidence),
    )
    return {
        "commands_dry_run": draft["commands"],
        "evidence_ids": [e["id"] for e in evidence["logs"][:5]],
        "change_ticket": open_ticket_draft(draft),  # ✅ human must submit
    }

Tie into OpenTelemetry for LLMs so suggestions carry trace IDs.

Approval UX that survives 3 a.m.

# slack modal — illustrative
# ✅ "Approve apply of rollout restart on payments-api (change CHG-9182)"
# ❌ "Auto-remediate" toggle enabled by default in prod

Closing checklist

✅ Dos
– ✅ Allowlist read-only CLIs in the tool layer
– ✅ Always attach dry-run variants in suggestions
– ✅ Cite logs/alarms/traces in every draft
– ✅ Dual-control for writes
– ✅ Audit execution separately from suggestion logs

❌ Don’ts
– ❌ Don’t give copilots standing write credentials
– ❌ Don’t auto-execute from chat emoji reactions alone without change IDs
– ❌ Don’t invent runbook steps without evidence
– ❌ Don’t skip cost controls during sev storms
– ❌ Don’t disable Guardrails “temporarily” on the incident channel

Related reading

• LLM Incident Runbooks: Ground On-Call Answers in CloudWatch Signals
• Claude Code Hooks: Gate Risky Shell Commands Before CI Runs
• OpenTelemetry for LLMs: Trace Prompt Latency Across Microservices
• LLM Cost Controls: Token Budgets Per PR and Per Engineer

Last updated on September 11, 2026