Self-Healing Tests: AI Quarantine With Hard Human Fix SLAs

“Self-healing” that silently deletes assertions is coverage rot with marketing. The unfair advantage is AI that quarantines flaky tests with a root-cause note, owner, and hard SLA — then fails the build if humans ignore the ticket past the deadline. Healing is triage plus accountability, not autopilot green.

⚡ TL;DR: Classify flake vs regression; quarantine only the flake class with @flaky + ticket; attach AI notes citing stacks and historical pass rates; enforce fix-or-delete SLA (e.g. 5 business days). Never auto-weaken assertions. Pair with CI Failure Triage Bots and AI Test Generation property tests.

Classify before quarantine

// ci/flake-classify.ts
export type Class = "flake" | "regression" | "infra" | "unknown";

export function classify(sig: {
  testId: string;
  passRate14d: number;
  changedInPR: boolean;
  sameStackAsKnownFlake: boolean;
}): Class {
  if (sig.changedInPR && sig.passRate14d > 0.95) return "regression"; // ✅ new failure on touched code
  if (sig.sameStackAsKnownFlake && sig.passRate14d < 0.85) return "flake";
  if (/ECONNRESET|runner|dockerd/.test(sig.testId)) return "infra";
  return "unknown";
}

❌ Quarantining anything that failed twice — that is how real regressions hide.

Quarantine artifact the bot must write

<!-- .quarantine/payments.spec.ts::retriesExhausted.md -->
test: payments.spec.ts > retriesExhausted
class: flake
owner: @payments-platform
sla_due: 2026-09-18
ai_notes: |
  Historical pass rate 71% (14d). Failure correlates with shared Redis
  fixture; stack matches flake#482. Not touched in this PR.
evidence:
  - ci_run: 889120
  - stack_hash: a1b2c3
action: quarantine_until_sla

// jest setup — illustrative
import { readFileSync, existsSync } from "node:fs";

function isQuarantined(testId: string): boolean {
  const p = `.quarantine/${testId}.md`;
  if (!existsSync(p)) return false;
  const due = /sla_due:\s*(\d{4}-\d{2}-\d{2})/.exec(readFileSync(p, "utf8"));
  if (!due) return false;
  // ✅ Past SLA → fail CI loudly (coverage rot prevention)
  if (new Date(due[1]) < new Date()) throw new Error(`quarantine_sla_breached:${testId}`);
  return true;
}

Human SLA is the product

Wire the same ownership maps you use for GraphRAG microservices so tickets land on the right team.

What AI may change vs never touch

# ✅ Allowed bot actions
# - open quarantine file + Jira/Linear ticket
# - suggest flake root cause from logs
# - re-run flake class once

# ❌ Forbidden
# - delete expect(...) / reduce timeouts “to green”
# - mark regression class as quarantine
# - mute entire suites

Closing checklist

✅ Dos
– ✅ Separate flake vs regression with pass-rate + PR touch signals
– ✅ Quarantine with owner, evidence, SLA date
– ✅ Fail CI on SLA breach
– ✅ Require CODEOWNER for permanent delete
– ✅ Re-run only the flake class automatically

❌ Don’ts
– ❌ Don’t auto-edit assertions to heal
– ❌ Don’t quarantine regressions on touched files
– ❌ Don’t let quarantine files live without due dates
– ❌ Don’t mute whole packages
– ❌ Don’t count quarantined tests as coverage wins

Related reading

• CI Failure Triage Bots: Separate Flaky Noise From Real Regressions
• AI Test Generation: Property Tests That Catch Race Conditions
• Deterministic Replay: Agent Sessions You Can Debug in Postmortems
• GraphRAG for Microservices: Call Graphs Beat Flat Chunk Retrieval

Last updated on September 11, 2026