5 views
ICMP or /health canaries stay green while login is broken, checkout 500s, or search returns empty hits. Continuous path verification runs scripted journeys against production-like environments every minute and burns error budgets when business paths fail—not when a single process answers HTTP 200.
⚡ TL;DR: Define critical journeys (auth, pay, search); run CloudWatch Synthetics / canaries every 1 minute from multiple regions; assert status, latency, and business invariants; alert on SLO burn. Pair with SLA Error Budgets, WAF Rate Rules for Mobile, and API Gateway WebSocket Fan-Out.
Continuous Path Verification: production guidance
text
/healthz green
+ auth canary fails (IdP redirect loop) → customers cannot log in
+ pay canary fails (processor 3DS timeout) → revenue stops
+ search canary fails (0 hits for known SKU) → discovery broken
Synthetics sketch (Puppeteer / Node)
javascript
const synthetics = require("Synthetics");
const log = require("SyntheticsLogger");
const flow = async function () {
const page = await synthetics.getPage();
await page.goto(process.env.BASE_URL + "/login", { waitUntil: "networkidle0" });
await page.type("#email", process.env.CANARY_USER);
await page.type("#password", process.env.CANARY_PASS);
await Promise.all([
page.waitForNavigation({ waitUntil: "networkidle0" }),
page.click("#submit"),
]);
const ok = await page.$("[data-testid=home-ready]");
if (!ok) throw new Error("auth_home_ready_missing");
// pay path: create intent for $0.00 test SKU, assert client_secret shape
const pay = await page.evaluate(async () => {
const r = await fetch("/v1/pay/intent", { method: "POST", credentials: "include" });
return { status: r.status, body: await r.json() };
});
if (pay.status !== 200 || !pay.body.clientSecret) throw new Error("pay_intent_failed");
// search: known fixture SKU must appear
await page.goto(process.env.BASE_URL + "/search?q=canary-sku-42");
const hit = await page.$("[data-sku='canary-sku-42']");
if (!hit) throw new Error("search_fixture_missing");
};
exports.handler = async () => synthetics.executeStep("path_verify", flow);
SLOs and burn alerts
| Journey | Availability SLO | Latency SLO | Alert |
|---|---|---|---|
| Auth | 99.9% / 30d | p95 < 2s | 1h burn > 2x |
| Pay | 99.95% / 30d | p95 < 3s | page immediately on 3 fails |
| Search | 99.5% / 30d | p95 < 1s | ticket on 15m burn |
yaml
Isolate canary users and test SKUs so WAF/rate limits do not block them unfairly—coordinate with WAF Rate Rules for Mobile. Tie burn to load shedding via SLA Error Budgets.
# CloudWatch alarm sketch
AlarmName: pay-canary-failed
MetricName: SuccessPercent
Namespace: CloudWatchSynthetics
Threshold: 100
ComparisonOperator: LessThanThreshold
EvaluationPeriods: 3
Period: 60
TreatMissingData: breaching
Closing checklist
- [ ] Auth, pay, and search journeys run every minute from ≥2 regions
- [ ] Assertions include business invariants, not only HTTP 200
- [ ] Canary credentials/SKUs isolated and rotated
- [ ] SLO burn alerts page humans with journey name + screenshot/HAR
- [ ] Deploys auto-hold when canaries fail in the target stage
- [ ]
/healthalone is never the only production signal
Related reading
- SLA Error Budgets: Drive Automated Load Shedding Before Hard Outages
- WAF Rate Rules for Mobile: Avoid Blocking Carrier-Grade NAT Users
- CloudWatch Logs Anomaly Detection: Ignore Deploy Noise, Catch Novel Errors
- Blue/Green Stateful Workers: Drain In-Flight Jobs Before Old Tasks Die
Watch: 100 Days of AI on YouTube
Short videos from CheatCoders, one AI engineering topic per day.
Most viewed
- Python Decorators Explained: From Simple Wrappers to Production Patterns
- Claude Code vs Cursor vs GitHub Copilot: Choose by Workflow, and Set Up One Repo That Works With All Three
- AI Agent Frameworks in 2025: LangGraph vs CrewAI vs AutoGen vs Raw API
- Python String Methods: Every str Method With Real Production Examples
- Css Animation Performance: Performance-Optimized Motion for Modern UIs
Newly added
- AWS Systems Manager Parameter Store: Hierarchical Runtime Config for Coding Agents
- Amazon Cognito Identity Pools: Federate Coding-Agent Tool Callers Without Long-Lived Keys
- Amazon Athena for Coding Agents: Query Tool Audit Logs and Eval Traces on S3 Without a Warehouse
- AWS Batch for Coding Agents: Parallel Overnight PR Reviews and Long-Running Jobs Without Lambda Timeouts
- Amazon Bedrock Prompt Management: Version and A/B Test Coding-Agent System Prompts
Deep-dive PDF
Get the expanded guide for this post — extra diagrams-style checklists, failure modes, and a production walkthrough. Free when you subscribe to CheatCoders.
Already subscribed? or open the subscribe page.
Discover more from CheatCoders
Subscribe to get the latest posts sent to your email.
