AI On-Call Copilot: A Project That Cannot Mutate Prod

⚡ TL;DR: IAM deny on mutating APIs for the agent role. Read-only tools only by default. propose_patch opens PRs/CRs. Break-glass assumes a separate role after two human approvals via Step Functions.

AI on-call copilot: production guidance

1. Runbook KB with Day 26 citations
2. Read-only CloudWatch/Logs tools
3. propose_patch → branch/PR only
4. Optional break-glass behind dual approval
5. Day 39 traces on every turn

IAM posture

{
  "Effect": "Deny",
  "Action": ["ecs:UpdateService", "eks:*", "ssm:SendCommand", "ec2:TerminateInstances", "rds:Delete*"],
  "Resource": "*"
}

def propose_patch(repo: str, path: str, diff: str, idem: str) -> dict:
    branch = f"oncall-bot/{idem[:8]}"
    gh.create_branch(repo, branch)
    gh.commit(repo, branch, path, diff, message="oncall suggestion")
    pr = gh.open_pr(repo, branch, title=f"On-call suggestion {idem[:8]}")
    return {"prUrl": pr.html_url, "applied": False}

Project idea: game day

Deliverables checklist

• [ ] Deny-by-default IAM for agent role
• [ ] Runbook RAG with fail-closed citations
• [ ] propose_patch → PR only
• [ ] Dual-control break-glass documented
• [ ] Traces on every turn
• [ ] Game-day script committed

Series navigation