Lambda X-Ray Sampling: Cost Versus Debuggability During Incidents

Always-on 100% X-Ray sampling is how tracing bills become a second cloud provider. Zero sampling is how Sev-1s become folklore. The unfair advantage is dynamic sampling: cheap baselines every day, automatic boosts during incidents, and hard expiry so “temporary” never becomes permanent.

⚡ TL;DR: Baseline 1–5% reservoir sampling per service. On incident open, flip a rule (or SSM flag) to 100% for the blasted service + dependencies for a TTL (30–120 min). Prefer tail sampling on errors when using OTel collectors. Pair traces with OpenTelemetry for LLMs and triage discipline from CI Failure Triage Bots.

Baseline rules that do not bankrupt you

{
  "rule_name": "checkout-baseline",
  "priority": 1000,
  "fixed_target": 1,
  "rate": 0.05,
  "service_name": "checkout-api",
  "service_type": "*",
  "host": "*",
  "http_method": "*",
  "url_path": "*",
  "resource_arn": "*",
  "attributes": {}
}

{
  "rule_name": "checkout-incident-boost",
  "priority": 10,
  "fixed_target": 100,
  "rate": 1.0,
  "service_name": "checkout-api",
  "resource_arn": "arn:aws:lambda:REGION:ACCT:function:checkout-api"
}

✅ Low priority number = evaluated first for incident boost.
❌ Leaving rate=1.0 on the baseline rule after the war room ends.

Automate boost + expiry

import { SSM } from "@aws-sdk/client-ssm";
import { XRay } from "@aws-sdk/client-xray";

export async function beginIncidentTraceBoost(fn: string, ttlMinutes = 60) {
  const until = new Date(Date.now() + ttlMinutes * 60_000).toISOString();
  await new SSM({}).putParameter({
    Name: `/tracing/boost/${fn}`,
    Value: until,
    Type: "String",
    Overwrite: true,
  });
  // Upsert high-priority sampling rule via XRay PutSamplingRules / console-as-code
}

export async function samplingRateFor(fn: string, baseline = 0.05): Promise<number> {
  try {
    const p = await new SSM({}).getParameter({ Name: `/tracing/boost/${fn}` });
    if (p.Parameter?.Value && Date.parse(p.Parameter.Value) > Date.now()) return 1;
  } catch { /* no boost */ }
  return baseline;
}

A EventBridge scheduler should delete expired boosts every five minutes — humans forget.

Error-biased fidelity without 100% forever

Inside Lambda, ensure AWS_XRAY_CONTEXT_MISSING=LOG_ERROR (or similar) so missing segments do not throw in warm paths.

Closing checklist

• [ ] Baseline sampling documented per service with monthly cost estimate
• [ ] Incident boost rule priority + TTL automation
• [ ] Scheduler clears expired boosts
• [ ] Error/tail sampling considered for rare failures
• [ ] Dashboards for trace ingest volume + $
• [ ] On-call runbook links boost toggle

Related reading

• OpenTelemetry for LLMs: Trace Prompt Latency Across Microservices
• CI Failure Triage Bots: Separate Flaky Noise From Real Regressions
• Lambda Warm Pools: Low-Latency Backends for Coding Agent Tools
• LLM Incident Runbooks: Ground On-Call Answers in CloudWatch Signals

Last updated on September 11, 2026