Day 40: Project: On-Call Copilot That Cannot Mutate Prod

PagerDuty at 3am is the wrong time to discover your agent can kubectl delete. This project ships a copilot that reads runbooks and metrics, proposes fixes, and cannot mutate production without dual control.

⚡ TL;DR: IAM deny on mutating APIs for the agent role. Read-only tools only by default. propose_patch opens PRs/CRs. Break-glass assumes a separate role after two human approvals via Step Functions.

Lab / project goals

1. Runbook KB with Day 26 citations
2. Read-only CloudWatch/Logs tools
3. propose_patch → branch/PR only
4. Optional break-glass behind dual approval
5. Day 39 traces on every turn

IAM posture

{
  "Effect": "Deny",
  "Action": ["ecs:UpdateService", "eks:*", "ssm:SendCommand", "ec2:TerminateInstances", "rds:Delete*"],
  "Resource": "*"
}

def propose_patch(repo: str, path: str, diff: str, idem: str) -> dict:
    branch = f"oncall-bot/{idem[:8]}"
    gh.create_branch(repo, branch)
    gh.commit(repo, branch, path, diff, message="oncall suggestion")
    pr = gh.open_pr(repo, branch, title=f"On-call suggestion {idem[:8]}")
    return {"prUrl": pr.html_url, "applied": False}

Project idea: game day

Inject a fake CPU alert. Require the copilot to cite the runbook span, pull the dashboard link, and open a PR changing only a feature-flag default — reviewed by a human. Time the dual-control break-glass path once without applying it to real prod.

Deliverables checklist

• [ ] Deny-by-default IAM for agent role
• [ ] Runbook RAG with fail-closed citations
• [ ] propose_patch → PR only
• [ ] Dual-control break-glass documented
• [ ] Traces on every turn
• [ ] Game-day script committed

Series navigation

← Day 39 · Day 41 →

Last updated September 11, 2026