OpenTelemetry Tail Sampling: Full Fidelity on Errors, Sparse on Success

Head sampling decides at the root span whether a trace lives—before you know it failed. Tail sampling waits until the trace completes, then keeps 100% of errors and slow requests while sparsely keeping healthy fast paths. That is how you stay useful under high RPS without drowning exporters.

⚡ TL;DR: Collect with a Collector pipeline that buffers traces, then apply policies: always_sample on status=ERROR, latency thresholds, and rare attributes; probabilistic sample the rest. Propagate consistent trace IDs. Pair with OpenTelemetry Sampling for Node, OpenTelemetry for LLMs, and X-Ray Service Maps Lie.

Head vs tail in one picture

Head sampling:
  ingress decides keep/drop immediately
  → cheap, but drops failures that look fine at start

Tail sampling:
  all spans land in Collector briefly
  → policy: keep errors + p99 latency + 1% success
  → drop the rest

At fifty thousand RPS you cannot keep everything; you also cannot afford to miss the only failing trace for a bad deploy.

Collector tail_sampling sketch

processors:
  tail_sampling:
    decision_wait: 10s
    num_traces: 100000
    expected_new_traces_per_sec: 20000
    policies:
      - name: errors
        type: status_code
        status_code: { status_codes: [ERROR] }
      - name: slow
        type: latency
        latency: { threshold_ms: 1500 }
      - name: checkout
        type: string_attribute
        string_attribute:
          key: http.route
          values: ["/v1/checkout", "/v1/pay"]
          enabled_regex_matching: false
      - name: probabilistic-rest
        type: probabilistic
        probabilistic: { sampling_percentage: 1 }

// App still records all spans locally; sampling decision is centralized
import { NodeSDK } from "@opentelemetry/sdk-node";
import { OTLPTraceExporter } from "@opentelemetry/exporter-trace-otlp-http";

const sdk = new NodeSDK({
  traceExporter: new OTLPTraceExporter({ url: process.env.OTEL_EXPORTER_OTLP_ENDPOINT }),
  // Prefer parent-based + Collector tail policies over aggressive SDK head drop
});

✅ Central Collector policies owned by platform.
❌ Each service invents incompatible head-sample ratios.

Cost and incident playbooks

During sev-1, temporarily raise probabilistic percentage or add an attribute policy for the suspected service—then revert. Same mindset as Lambda X-Ray Sampling. Watch Collector memory: decision_wait and num_traces are knobs that OOM if undersized.

For LLM tool chains, keep error spans and high token-latency traces—see OpenTelemetry for LLMs. Fix missing edges so tail policies see complete traces (X-Ray Service Maps Lie).

Closing checklist

• [ ] Collector tail_sampling keeps ERROR and high-latency traces at 100%
• [ ] Success paths use low probabilistic sample (e.g. 0.5–2%)
• [ ] Critical routes can force-keep via attribute policies
• [ ] decision_wait / buffer sized for peak RPS without OOM
• [ ] Incident runbook includes temporary sample bump + revert
• [ ] Trace context propagates across Node, Lambda, and LLM tools

Related reading

• OpenTelemetry Sampling for Node: Stay Useful at Fifty Thousand RPS
• OpenTelemetry for LLMs: Trace Prompt Latency Across Microservices
• X-Ray Service Maps Lie: Fix Missing Spans in Node Trace Edges
• Lambda X-Ray Sampling: Cost Versus Debuggability During Incidents

Last updated on September 11, 2026