Day 82: Voice and Realtime Agents

Voice agents look demos-great and ship-poor: turn-taking, barge-in, noise, and “please don’t make me talk to a bot in an open office.” Use voice when hands-free latency wins; otherwise keep text as default.

⚡ TL;DR: Design interruptible sessions. Separate STT partials from agent commits. Confirm irreversible tools by voice and UI. Offer mute / text fallback always.

Turn-taking mechanics

# ✅ Barge-in: cancel TTS and freeze tool commits
class VoiceSession:
    def __init__(self):
        self.speaking = False
        self.pending_tool = None

    def on_user_speech_start(self):
        self.speaking = False
        cancel_tts()
        if self.pending_tool and not self.pending_tool.committed:
            self.pending_tool.cancel()

❌ Firing deploy_prod because STT heard “deploy” in “don’t deploy.”

When not to use voice

• Deep code review, long diffs, compliance attestations.
• Shared spaces without headsets.
• Anything requiring copy-pasteable artifacts (prefer text + links).

Failure modes

Echo cancellation and duplex bugs cause the agent to hear itself and loop. Half-duplex with clear UX beats broken full-duplex. Retaining raw audio longer than the incident window creates DSAR nightmares — default retention hours, not months.

Closing checklist

• [ ] Interrupt / barge-in tested
• [ ] Tools gated on final transcript + confidence
• [ ] Text fallback one tap away
• [ ] Redact audio retention by default
• [ ] Explicit non-goals documented for PM

Series navigation

Day 81: Long-Term Memory Without Becoming Creepy · Day 83: Multimodal Coding: Screenshots of Broken UIs

Last updated September 11, 2026