Differential Testing: AI Oracles Validated Against Shadow Traffic

AI refactors compile and pass unit tests while quietly changing auth headers, rounding money, or reordering JSON keys that clients hash. Differential testing against shadow production traffic is how you catch behavioral drift before cutover—treat the live service as the oracle, the AI candidate as the suspect.

⚡ TL;DR: Mirror a slice of prod requests to the candidate; compare status, headers you care about, and semantic body equality (not byte equality). Quarantine diffs; never auto-cut over on “mostly green.” Pair with AI Migration Assistants: Strangler Fig Cuts and Semantic Diff Review.

Shadow fan-out without doubling write load

// gateway/shadow.ts
export async function shadowCompare(
  req: Request,
  primary: Handler,
  candidate: Handler,
  opts: { sampleRate: number; compareWrites: false }
) {
  const primaryRes = await primary(req);
  if (Math.random() > opts.sampleRate) return primaryRes;
  if (req.method !== "GET" && !opts.compareWrites) {
    // ❌ Never shadow POST/PUT to a candidate that mutates shared DBs
    return primaryRes;
  }
  // fire-and-forget candidate
  void candidate(req.clone()).then((cand) =>
    enqueueDiff({ primary: primaryRes, cand, path: req.url })
  );
  return primaryRes;
}

✅ Read-only shadow by default.
❌ Shadowing checkout POSTs into a shared DynamoDB table.

Semantic matchers beat deep equality

import { createHash } from "node:crypto";

type Diff = { field: string; primary: unknown; candidate: unknown };

export function semanticDiff(a: unknown, b: unknown, path = "$"): Diff[] {
  const out: Diff[] = [];
  if (typeof a !== typeof b) {
    out.push({ field: path, primary: a, candidate: b });
    return out;
  }
  if (Array.isArray(a) && Array.isArray(b)) {
    // order-insensitive for unordered collections
    const sa = [...a].map(stable).sort();
    const sb = [...b].map(stable).sort();
    if (JSON.stringify(sa) !== JSON.stringify(sb)) {
      out.push({ field: path, primary: a, candidate: b });
    }
    return out;
  }
  if (a && typeof a === "object") {
    const ka = Object.keys(a as object).sort();
    const kb = Object.keys(b as object).sort();
    for (const k of new Set([...ka, ...kb])) {
      out.push(...semanticDiff((a as any)[k], (b as any)[k], `${path}.${k}`));
    }
    return out;
  }
  // money: compare cents; timestamps: ignore
  if (path.endsWith(".updatedAt")) return out;
  if (a !== b) out.push({ field: path, primary: a, candidate: b });
  return out;
}

function stable(v: unknown) {
  return createHash("sha256").update(JSON.stringify(v)).digest("hex");
}

Ignore fields the AI was allowed to change (new telemetry headers). Fail hard on authorization, totals, and idempotency keys.

Promotion criteria

Wire promotion to the same human gates you use in Step Functions Multi-Agent Pipelines.

Closing checklist

✅ Dos
– ✅ Shadow GET/HEAD (and safe idempotent reads) only unless isolated stores
– ✅ Semantic matchers with allowlisted ignore paths
– ✅ 24h burn-in metrics before cutover
– ✅ Store failing request IDs for replay

❌ Don’ts
– ❌ Don’t byte-compare timestamps or request IDs
– ❌ Don’t auto-cut over from an agent
– ❌ Don’t shadow writes into shared production data
– ❌ Don’t treat “unit tests green” as behavioral equivalence

Related reading

• AI Migration Assistants: Strangler Fig Cuts With Contract Tests
• Semantic Diff Review: AI That Flags Behavioral Drift Only
• Spec-First AI Development: OpenAPI Remains the Only Source of Truth
• Evaluating AI Coding Tools: Blind A/B Tests on Real Tickets

Last updated on September 11, 2026