Claude Computer Use: Capture Evidence From Flaky UI Test Failures

“Fails on CI, passes locally” dies when the agent can reproduce the UI path, dump DOM + HAR, and attach artifacts to the ticket. Computer-use agents are not for auto-healing production—they are for evidence collection at the speed of a flaky suite.

⚡ TL;DR: On Playwright/Cypress failure, spawn a sandboxed computer-use session with the failing URL + seed; capture screenshot, accessibility tree, network HAR, and console errors; upload to the job artifacts and Jira. Never grant prod credentials. Pair with CI Failure Triage Bots and Self-Healing Tests Quarantine.

Failure hook → evidence agent

// ci/on-ui-fail.ts
export async function captureEvidence(fail: {
  testName: string;
  traceUrl: string;
  videoUrl?: string;
  seed: string;
}) {
  const prompt = [
    `Reproduce flaky UI test: ${fail.testName}`,
    `Open staging URL from seed ${fail.seed}`,
    `Do NOT change application data. Read-only.`,
    `Return: steps, console errors, selector that flake, HAR path.`,
  ].join("\n");

  // Computer-use runs in ephemeral ECS — see secure sandboxes
  const session = await startComputerUseAgent({
    prompt,
    allowlistDomains: ["staging.example.com"],
    denyWrites: true,
    maxMinutes: 8,
  });
  return session.artifacts; // png, har, a11y.json
}

✅ Domain allowlist + read-only instructions.
❌ Agent with prod SSO and write access “to fix the button.”

Artifact contract for tickets

## Flake evidence (auto)
- Test: `checkout.spec.ts:141`
- Screenshot: `s3://ci-artifacts/.../fail.png`
- HAR: `.../network.har`
- Console: 3 errors (first: TypeError cannot read 'id' of undefined)
- Suspected selector: `[data-testid=pay-cta]` intermittent detach
- Agent session: `cu-2026-09-11-ab12`

Sanitize tickets before agents read them—tracker text is untrusted (Prompt Injection in Trackers).

Quarantine, don’t infinite-retry

# After N evidence-backed flakes → quarantine with SLA
if: flake_count >= 3
steps:
  - run: pnpm tsx scripts/quarantine-test.ts --sla-days=5
  - run: gh issue create --title "Flake: ${TEST}" --body-file evidence.md

Computer-use cost adds up—sample only failures that already failed twice, or gate on labels. Metrics mindset from AI Pair Programming Metrics: evidence quality over raw retry count.

Closing checklist

• [ ] Computer-use only on staging + allowlisted domains
• [ ] Artifacts: screenshot, HAR, console, a11y tree
• [ ] No prod credentials in the agent environment
• [ ] Ticket body includes session ID for replay
• [ ] Quarantine SLA after repeated flakes; no silent skip forever
• [ ] Sanitize tracker fields before agent consumption

Related reading

• CI Failure Triage Bots: Separate Flaky Noise From Real Regressions
• Self-Healing Tests: AI Quarantine With Hard Human Fix SLAs
• Prompt Injection in Trackers: Sanitize Jira Before Agents Read Issues
• Secure AI Sandboxes: Ephemeral ECS Tasks for Agent Tool Execution

Last updated on September 11, 2026