Day 16: Streaming UX Without Lying About Completeness

Streaming makes agents feel alive — and makes it easy to lie. Day 16 covers honest streaming UX: partial tokens, explicit tool-pause states, completion vs cancel semantics, and backpressure when API Gateway, Lambda response streaming, or WebSockets meet slow clients.

⚡ TL;DR: Stream tokens for prose; buffer structured tool args until valid. Emit explicit events (token, tool_start, tool_result, final, error). Do not mark success until validators pass. Apply read backpressure; drop or disconnect abusive clients.

Event protocol beats raw text

type Ev =
  | { type: "token"; text: string }
  | { type: "tool_start"; name: string }
  | { type: "tool_result"; name: string; ok: boolean }
  | { type: "final"; answer: string; cited?: string[] }
  | { type: "error"; code: string };

❌ Concatenating tool JSON into the same text stream users read as English, then parsing it back with regret.

When a tool runs, pause token streaming and show “Running run_tests…”. After results, resume. Users should never think the answer is complete while a tool is in flight.

Completeness and cancellation

Define:

• final — validators passed; safe to persist.
• error — failed closed; show reason.
• cancelled — client disconnected; abort tools if possible.

Do not auto-apply patches from a stream that never emitted final.

Backpressure on AWS edges

Lambda response streaming and API Gateway have idle timeouts and payload limits. For longer agent sessions, prefer WebSockets (API GW WS) or HTTP/2 with heartbeats. If the client stops reading, stop generating — continuing burns money (Day 15) and fills buffers.

# ✅ Cooperative cancel
if client_gone.is_set():
    cancel_tools()
    raise GenerationCancelled()

Closing checklist

• [ ] Typed stream events in the client SDK
• [ ] Tool pauses visible in UI
• [ ] final only after validation
• [ ] Cancel path stops work and billing
• [ ] Heartbeats / WS for long sessions
• [ ] Load test slow clients

Worked example: structured patch streaming

Buffer NDJSON hunk frames (or full tool args) until schema-valid, then emit tool_start. Optionally stream a preview channel marked non-authoritative. Never write files from the preview channel.

Failure modes to watch

• Success toasts on HTTP 200 while stream still open.
• No heartbeats → idle timeouts mid-answer.
• Unbounded server buffers for stalled browsers.
• Applying partial JSON tool calls.

Field notes from production

Clients must render error and cancelled distinctly from truncated networks. Mobile apps that reconnect should resume with Last-Event-ID or a session cursor, not duplicate tool side effects — tie to idempotency keys (Day 11).

Implementation sketch

// Implementation sketch: final only after validate
for await (const ev of stream) {
  if (ev.type === "token") ui.append(ev.text);
  if (ev.type === "final" && validate(ev)) ui.complete(ev);
}

Operator addendum

Contract-test your client SDK against a recorded event sequence including error and mid-tool disconnects. UI bugs here look like ‘AI flakiness’ in user reports.

Testing stream clients

Use a mock server that emits tokens, pauses for a fake tool, then error. Assert UIs do not show success. Add a test where final arrives with invalid citations and ensure the client displays refuse. Mobile networks will drop mid-stream; verify resume does not double-apply tools thanks to idempotency keys. These tests are dull and more valuable than another screenshot of tokens flying.

Extended discussion

Return to the core angle for Day 16: Partial tokens, tool pauses, and backpressure on API Gateway/Lambda. That sentence is the acceptance lens for every design review this week. If a proposed change does not make this angle easier to measure or enforce, it is a distraction.

Write down three metrics you will look at after shipping Day 16 ideas, schedule a 45-minute readout, and archive the notes next to the eval artifacts. Architecture without a readout becomes slideshow archaeology.

Pair this day with the adjacent lessons in the series navigation below. Forward links exist so you can keep momentum; backward links exist so you can repair foundations when a later lab fails for boring earlier reasons.

Practically, allocate half a day to implement the smallest vertical slice, half a day to wire measurement, and refuse to polish UI until both are done. This ordering is how bootcamp projects stay honest under time pressure.

Revisit assumptions whenever the model ID, embedding ID, or index alias changes — treat those as breaking changes for Day 16 behaviors, with the same seriousness as a database migration. Canary first, then promote.

Revisit assumptions whenever the model ID, embedding ID, or index alias changes — treat those as breaking changes for Day 16 behaviors, with the same seriousness as a database migration. Canary first, then promote.

Revisit assumptions whenever the model ID, embedding ID, or index alias changes — treat those as breaking changes for Day 16 behaviors, with the same seriousness as a database migration. Canary first, then promote.

Revisit assumptions whenever the model ID, embedding ID, or index alias changes — treat those as breaking changes for Day 16 behaviors, with the same seriousness as a database migration. Canary first, then promote.

Revisit assumptions whenever the model ID, embedding ID, or index alias changes — treat those as breaking changes for Day 16 behaviors, with the same seriousness as a database migration. Canary first, then promote.

Revisit assumptions whenever the model ID, embedding ID, or index alias changes — treat those as breaking changes for Day 16 behaviors, with the same seriousness as a database migration. Canary first, then promote.

Series navigation

← Day 15 · Day 17 →

Last updated September 11, 2026