CI Failure Triage Bots: Separate Flaky Noise From Real Regressions

Not every red X deserves a page. A triage bot that cannot tell flake from regression will either spam on-call or silently rerun real breakages until the deploy train derails. Classify failures with signatures from history, auto-rerun only the flake class, and escalate novel signatures hard.

⚡ TL;DR: Fingerprint failures (test id + error class + stack top frames). Train/maintain a classifier on labeled history: flake | regression | infra | novel. Auto-rerun flakes with a budget; never auto-rerun regression or novel. Page on novel post-deploy signatures. Pair with AI Test Generation: Property Tests and OpenTelemetry for LLMs for correlating CI with prod.

Fingerprints beat raw log dumps

import { createHash } from "node:crypto";

export function fingerprint(fail: {
  testId: string;
  message: string;
  stack: string[];
}): string {
  const top = fail.stack.slice(0, 3).map(normalizeFrame).join("|");
  const msgClass = fail.message.replace(/\d+/g, "N").replace(/0x[0-9a-f]+/gi, "HEX");
  return createHash("sha256")
    .update(`${fail.testId}::${msgClass}::${top}`)
    .digest("hex")
    .slice(0, 16);
}

function normalizeFrame(f: string) {
  return f.replace(/:\d+:\d+/g, ":L:C").replace(/\/home\/runner\/.*/g, "REPO");
}

✅ Stable across line-number jitter and runner paths.
❌ Hashing the entire log (unique every time → everything looks novel).

Classifier with an explicit novel bucket

type Label = "flake" | "regression" | "infra" | "novel";

export async function classify(fp: string, ctx: {
  seenCount90d: number;
  flakeRatio90d: number;
  firstSeenAfterDeploy: boolean;
  matchedInfraSignature: boolean;
}): Promise<Label> {
  if (ctx.matchedInfraSignature) return "infra";
  if (ctx.seenCount90d === 0) return "novel";
  if (ctx.flakeRatio90d >= 0.6 && !ctx.firstSeenAfterDeploy) return "flake";
  if (ctx.firstSeenAfterDeploy && ctx.flakeRatio90d < 0.2) return "regression";
  // LLM assist for ambiguous mid-band — optional, never sole authority
  return "novel"; // fail closed toward human eyes
}

Use historical CI data; an LLM can narrate but must not be the only voter on auto-rerun. Ground narratives like LLM Incident Runbooks.

Actions by class

flake:
  action: rerun_job
  max_reruns: 1
  notify: silent_metric
regression:
  action: block_merge
  notify: pr_comment + slack_team
infra:
  action: rerun_on_new_runner
  notify: platform_oncall (if burst)
novel:
  action: block_merge
  notify: page_deploying_team
  require: human_label_within_24h

export async function handleFailure(label: Label, job: Job) {
  switch (label) {
    case "flake":
      if (job.reruns < 1) return rerun(job);
      return quarantineTest(job.testId, { slaHours: 72 });
    case "regression":
      return blockMerge(job.pr, `regression_fingerprint=${job.fp}`);
    case "infra":
      return rerun(job, { runnerLabel: "fresh" });
    case "novel":
      await page(job.deployingTeam, job);
      return blockMerge(job.pr, `novel_fingerprint=${job.fp}`);
  }
}

Quarantine with a hard fix SLA — do not let flakes rot coverage forever.

Feedback loop so the classifier stays honest

Track precision/recall weekly; alert if flake auto-rerun rate spikes (might be mislabeling regressions).

Closing checklist

• [ ] Fingerprints from test id + normalized message + top stack frames
• [ ] Labels include explicit novel; unknown ≠ flake
• [ ] Auto-rerun only for flake (budget 1) and controlled infra
• [ ] regression and novel block merge and notify humans
• [ ] Quarantine SLA for repeated flakes; metrics on misclassification
• [ ] Post-deploy novel signatures page the deploying team

Related reading

• AI Test Generation: Property Tests That Catch Race Conditions
• LLM Incident Runbooks: Ground On-Call Answers in CloudWatch Signals
• Semantic Diff Review: AI That Flags Behavioral Drift Only
• OpenTelemetry for LLMs: Trace Prompt Latency Across Microservices

Last updated on September 11, 2026