5 views
Always-on 100% X-Ray sampling is how tracing bills become a second cloud provider. Zero sampling is how Sev-1s become folklore. The unfair advantage is dynamic sampling: cheap baselines every day, automatic boosts during incidents, and hard expiry so “temporary” never becomes permanent.
⚡ TL;DR: Baseline 1–5% reservoir sampling per service. On incident open, flip a rule (or SSM flag) to 100% for the blasted service + dependencies for a TTL (30–120 min). Prefer tail sampling on errors when using OTel collectors. Pair traces with OpenTelemetry for LLMs and triage discipline from CI Failure Triage Bots.
Lambda X-Ray Sampling: production guidance
json
{
"rule_name": "checkout-baseline",
"priority": 1000,
"fixed_target": 1,
"rate": 0.05,
"service_name": "checkout-api",
"service_type": "*",
"host": "*",
"http_method": "*",
"url_path": "*",
"resource_arn": "*",
"attributes": {}
}
json
✅ Low priority number = evaluated first for incident boost.{
"rule_name": "checkout-incident-boost",
"priority": 10,
"fixed_target": 100,
"rate": 1.0,
"service_name": "checkout-api",
"resource_arn": "arn:aws:lambda:REGION:ACCT:function:checkout-api"
}
❌ Leaving rate=1.0 on the baseline rule after the war room ends.
Automate boost + expiry
typescript
A EventBridge scheduler should delete expired boosts every five minutes — humans forget.
import { SSM } from "@aws-sdk/client-ssm";
import { XRay } from "@aws-sdk/client-xray";
export async function beginIncidentTraceBoost(fn: string, ttlMinutes = 60) {
const until = new Date(Date.now() + ttlMinutes * 60_000).toISOString();
await new SSM({}).putParameter({
Name: `/tracing/boost/${fn}`,
Value: until,
Type: "String",
Overwrite: true,
});
// Upsert high-priority sampling rule via XRay PutSamplingRules / console-as-code
}
export async function samplingRateFor(fn: string, baseline = 0.05): Promise<number> {
try {
const p = await new SSM({}).getParameter({ Name: `/tracing/boost/${fn}` });
if (p.Parameter?.Value && Date.parse(p.Parameter.Value) > Date.now()) return 1;
} catch { /* no boost */ }
return baseline;
}
Error-biased fidelity without 100% forever
| Approach | Pros | Cons |
|---|---|---|
| Head sample 5% | Cheap, simple | Miss rare bugs |
| Incident 100% TTL | Perfect for Sev-1 | Needs automation |
| Tail sample errors in collector | High signal | Extra infra |
| Powertools + ADOT selective export | Flexible | Config complexity |
AWS_XRAY_CONTEXT_MISSING=LOG_ERROR (or similar) so missing segments do not throw in warm paths.
Closing checklist
- [ ] Baseline sampling documented per service with monthly cost estimate
- [ ] Incident boost rule priority + TTL automation
- [ ] Scheduler clears expired boosts
- [ ] Error/tail sampling considered for rare failures
- [ ] Dashboards for trace ingest volume + $
- [ ] On-call runbook links boost toggle
Related reading
- OpenTelemetry for LLMs: Trace Prompt Latency Across Microservices
- CI Failure Triage Bots: Separate Flaky Noise From Real Regressions
- Lambda Warm Pools: Low-Latency Backends for Coding Agent Tools
- LLM Incident Runbooks: Ground On-Call Answers in CloudWatch Signals
Watch: 100 Days of AI on YouTube
Short videos from CheatCoders, one AI engineering topic per day.
Most viewed
- Python Decorators Explained: From Simple Wrappers to Production Patterns
- Claude Code vs Cursor vs GitHub Copilot: Choose by Workflow, and Set Up One Repo That Works With All Three
- AI Agent Frameworks in 2025: LangGraph vs CrewAI vs AutoGen vs Raw API
- Python String Methods: Every str Method With Real Production Examples
- Css Animation Performance: Performance-Optimized Motion for Modern UIs
Newly added
- AWS Systems Manager Parameter Store: Hierarchical Runtime Config for Coding Agents
- Amazon Cognito Identity Pools: Federate Coding-Agent Tool Callers Without Long-Lived Keys
- Amazon Athena for Coding Agents: Query Tool Audit Logs and Eval Traces on S3 Without a Warehouse
- AWS Batch for Coding Agents: Parallel Overnight PR Reviews and Long-Running Jobs Without Lambda Timeouts
- Amazon Bedrock Prompt Management: Version and A/B Test Coding-Agent System Prompts
Deep-dive PDF
Get the expanded guide for this post — extra diagrams-style checklists, failure modes, and a production walkthrough. Free when you subscribe to CheatCoders.
Already subscribed? or open the subscribe page.
Discover more from CheatCoders
Subscribe to get the latest posts sent to your email.
