Continuous Path Verification: Minute Canaries for Auth, Pay, and Search

ICMP or /health canaries stay green while login is broken, checkout 500s, or search returns empty hits. Continuous path verification runs scripted journeys against production-like environments every minute and burns error budgets when business paths fail—not when a single process answers HTTP 200.

⚡ TL;DR: Define critical journeys (auth, pay, search); run CloudWatch Synthetics / canaries every 1 minute from multiple regions; assert status, latency, and business invariants; alert on SLO burn. Pair with SLA Error Budgets, WAF Rate Rules for Mobile, and API Gateway WebSocket Fan-Out.

Journeys beat pings

/healthz green
  + auth canary fails (IdP redirect loop)     → customers cannot log in
  + pay canary fails (processor 3DS timeout) → revenue stops
  + search canary fails (0 hits for known SKU) → discovery broken

Synthetics sketch (Puppeteer / Node)

const synthetics = require("Synthetics");
const log = require("SyntheticsLogger");

const flow = async function () {
  const page = await synthetics.getPage();
  await page.goto(process.env.BASE_URL + "/login", { waitUntil: "networkidle0" });
  await page.type("#email", process.env.CANARY_USER);
  await page.type("#password", process.env.CANARY_PASS);
  await Promise.all([
    page.waitForNavigation({ waitUntil: "networkidle0" }),
    page.click("#submit"),
  ]);
  const ok = await page.$("[data-testid=home-ready]");
  if (!ok) throw new Error("auth_home_ready_missing");

  // pay path: create intent for $0.00 test SKU, assert client_secret shape
  const pay = await page.evaluate(async () => {
    const r = await fetch("/v1/pay/intent", { method: "POST", credentials: "include" });
    return { status: r.status, body: await r.json() };
  });
  if (pay.status !== 200 || !pay.body.clientSecret) throw new Error("pay_intent_failed");

  // search: known fixture SKU must appear
  await page.goto(process.env.BASE_URL + "/search?q=canary-sku-42");
  const hit = await page.$("[data-sku='canary-sku-42']");
  if (!hit) throw new Error("search_fixture_missing");
};
exports.handler = async () => synthetics.executeStep("path_verify", flow);

SLOs and burn alerts

# CloudWatch alarm sketch
AlarmName: pay-canary-failed
MetricName: SuccessPercent
Namespace: CloudWatchSynthetics
Threshold: 100
ComparisonOperator: LessThanThreshold
EvaluationPeriods: 3
Period: 60
TreatMissingData: breaching

Isolate canary users and test SKUs so WAF/rate limits do not block them unfairly—coordinate with WAF Rate Rules for Mobile. Tie burn to load shedding via SLA Error Budgets.

Closing checklist

• [ ] Auth, pay, and search journeys run every minute from ≥2 regions
• [ ] Assertions include business invariants, not only HTTP 200
• [ ] Canary credentials/SKUs isolated and rotated
• [ ] SLO burn alerts page humans with journey name + screenshot/HAR
• [ ] Deploys auto-hold when canaries fail in the target stage
• [ ] /health alone is never the only production signal

Related reading

• SLA Error Budgets: Drive Automated Load Shedding Before Hard Outages
• WAF Rate Rules for Mobile: Avoid Blocking Carrier-Grade NAT Users
• CloudWatch Logs Anomaly Detection: Ignore Deploy Noise, Catch Novel Errors
• Blue/Green Stateful Workers: Drain In-Flight Jobs Before Old Tasks Die

Last updated on September 11, 2026