Day 83: Multimodal Coding: Screenshots of Broken UIs

Screenshots of broken UIs are gold for multimodal coding agents — and a PII leak magnet. Treat pixels as untrusted evidence: redact, bound, and attach as artifacts with retention rules.

⚡ TL;DR: Redact emails, tokens, faces, and customer data before the model sees the image. Store originals in a locked evidence bucket. Prefer DOM/accessibility trees when you can capture them instead of raw pixels.

Evidence pipeline

# ✅ Redact then caption — model never sees raw prod screenshot
def prepare_ui_evidence(png: bytes, meta: dict) -> dict:
    redacted = redact_pii_image(png)          # emails, JWTs, avatars
    thumb = resize_max(redacted, 1280)
    caption = vision_caption(thumb)           # optional
    evidence_id = store_evidence(raw=png, redacted=thumb, meta=meta)
    return {"evidence_id": evidence_id, "caption": caption, "redacted": True}

What the agent should receive

1. Redacted image (or DOM snapshot JSON).
2. Route / build SHA / browser version.
3. Expected vs actual short text from the reporter.

❌ Pasting full production admin screens into a consumer multimodal API without DPA and redaction.

Failure modes

OCR of redacted images that reconstructs emails from partial glyphs — re-check OCR output with the same PII filters. Designers pasting Figma screenshots with customer data into Slack threads that the bot scrapes — extend redaction to chat ingest.

Closing checklist

• [ ] Automated PII redaction before model call
• [ ] Evidence bucket with IAM + retention
• [ ] Prefer accessibility tree when available
• [ ] Eval set of UI bugs with redacted fixtures
• [ ] Ban raw screenshot paste in chat policy

Series navigation

Day 82: Voice and Realtime Agents · Day 84: SQL and Warehouse Copilots

Last updated September 11, 2026