Day 17: Human-in-the-Loop Gates That Don’t Stall

Production-touching agents need humans without turning the orchestrator into a frozen HTTP request. Day 17 designs HITL gates as asynchronous dual-control: the agent proposes, a queue holds intent, EventBridge/Slack collects approvals, and execution resumes with an audit trail — not a 15-minute blocking modal inside Lambda.

⚡ TL;DR: For risky tools, enqueue PendingAction with blast radius. Notify approvers. Resume on Approved/Rejected events with TTL. Never hold a synchronous agent request open waiting for a human.

Sync vs async approval

# ✅ Enqueue, do not block
def request_prod_tool(action: dict, actor: str) -> str:
    pid = ulid()
    db.put(PendingAction(pid, action, actor, status="PENDING", ttl=hours(24)))
    bus.put_events([{"DetailType": "ApprovalNeeded", "Detail": {"pid": pid}}])
    return pid

❌ input("Approve?") semantics inside a Lambda serving API Gateway.

Dual control details

• Two-person rule for high blast radius (prod data deletes, IAM widens).
• Approver ≠ proposer when policy requires.
• Show diff, blast radius, and idempotency key in the approval card.
• On approve, executor verifies the payload hash still matches — prevent swap attacks.

Wire Slack/Teams via EventBridge → Lambda → chat API; listen for button callbacks to emit Approved.

Agent UX

The agent receives status=waiting_for_approval and can continue other safe work or park the session. Session memory (Day 13) stores pending_pid. Streaming UX (Day 16) emits tool_start with awaiting_human=true.

Closing checklist

• [ ] Risky tools marked in registry
• [ ] PendingAction state machine with TTL
• [ ] Dual-control policy documented
• [ ] Payload hash binding on approve
• [ ] Audit log: who proposed, who approved, what ran
• [ ] No synchronous human wait in request workers

Worked example: prod alarm deletion

Agent proposes delete_alarms. System enqueues, pages on-call. Approver sees alarm names + region. On approve, executor deletes with the same idempotency key. On TTL expire, mark rejected and notify proposer.

Failure modes to watch

• Approve button without hash check (TOCTOU).
• Same user approving when dual control required.
• Infinite pending without TTL.
• Silent execute if notification fails — fail closed instead.

Field notes from production

Approval fatigue is real. Risk-score actions so low-blast changes auto-run and high-blast always HITL. Rotate approvers; measure median time-to-approve. If approvals take hours, agents will be abandoned — fix the path, do not remove the gate quietly.

Implementation sketch

# Implementation sketch: approve handler
def on_approve(pid, approver, payload_hash):
    p = db.get(pid)
    if p.hash != payload_hash: raise Tamper()
    if approver == p.actor and requires_dual(p): raise DualControl()
    execute(p.action, idem=p.idem)

Operator addendum

Store rejection reasons. ‘No’ without feedback trains proposers to spam. Structured reject codes improve future agent plans and human trust.

Risk scoring actions

Assign blast radius scores: read=0, draft PR=1, merge=3, prod IAM=5. Auto-approve ≤1 in business hours for trusted bots; always HITL ≥3. Make scores visible in the approval UI. Revisit scores after incidents. The goal is scarce human attention on the dangerous tail, not theater approvals on every lint fix.

Extended discussion

Return to the core angle for Day 17: Dual control for prod tools; queue + EventBridge, not a blocking modal. That sentence is the acceptance lens for every design review this week. If a proposed change does not make this angle easier to measure or enforce, it is a distraction.

Write down three metrics you will look at after shipping Day 17 ideas, schedule a 45-minute readout, and archive the notes next to the eval artifacts. Architecture without a readout becomes slideshow archaeology.

Pair this day with the adjacent lessons in the series navigation below. Forward links exist so you can keep momentum; backward links exist so you can repair foundations when a later lab fails for boring earlier reasons.

Practically, allocate half a day to implement the smallest vertical slice, half a day to wire measurement, and refuse to polish UI until both are done. This ordering is how bootcamp projects stay honest under time pressure.

Revisit assumptions whenever the model ID, embedding ID, or index alias changes — treat those as breaking changes for Day 17 behaviors, with the same seriousness as a database migration. Canary first, then promote.

Revisit assumptions whenever the model ID, embedding ID, or index alias changes — treat those as breaking changes for Day 17 behaviors, with the same seriousness as a database migration. Canary first, then promote.

Revisit assumptions whenever the model ID, embedding ID, or index alias changes — treat those as breaking changes for Day 17 behaviors, with the same seriousness as a database migration. Canary first, then promote.

Revisit assumptions whenever the model ID, embedding ID, or index alias changes — treat those as breaking changes for Day 17 behaviors, with the same seriousness as a database migration. Canary first, then promote.

Revisit assumptions whenever the model ID, embedding ID, or index alias changes — treat those as breaking changes for Day 17 behaviors, with the same seriousness as a database migration. Canary first, then promote.

Revisit assumptions whenever the model ID, embedding ID, or index alias changes — treat those as breaking changes for Day 17 behaviors, with the same seriousness as a database migration. Canary first, then promote.

Series navigation

← Day 16 · Day 18 →

Last updated September 11, 2026