OpenTelemetry Sampling for Node: Stay Useful at Fifty Thousand RPS

At 50k RPS, 100% traces are not observability — they are a denial-of-wallet attack against your collector. Head sampling alone drops errors you needed; no sampling melts X-Ray/OTLP exporters and Node event loops under protobuf encode. The unfair advantage is layered policy: low head sample for success, guaranteed keep for errors/high latency via tail sampling (or parent-based rules), and exporter backpressure that sheds before the API does.

⚡ TL;DR: Start with parent-based head sampler at 1–5% for healthy traffic; force-sample error and >p99 latency traces; run a tail sampler (Gateway/collector) for final decisions; bound queue + batch in the Node SDK. Pair with OpenTelemetry Tail Sampling and OpenTelemetry for LLMs.

Head sample without blinding incidents

// src/otel.ts
import { NodeSDK } from "@opentelemetry/sdk-node";
import { ParentBasedSampler, TraceIdRatioBasedSampler, SamplingDecision } from "@opentelemetry/sdk-trace-base";
import { BatchSpanProcessor } from "@opentelemetry/sdk-trace-base";
import { OTLPTraceExporter } from "@opentelemetry/exporter-trace-otlp-http";

class ErrorAwareRatioSampler extends TraceIdRatioBasedSampler {
  // Ratio applied only when parent says "ask child"
  constructor(private ratio: number) { super(ratio); }
}

const exporter = new OTLPTraceExporter({ url: process.env.OTLP_URL });
const processor = new BatchSpanProcessor(exporter, {
  maxQueueSize: 2048,          // ✅ bound memory
  scheduledDelayMillis: 5000,
  maxExportBatchSize: 512,
});

const sdk = new NodeSDK({
  sampler: new ParentBasedSampler({
    root: new TraceIdRatioBasedSampler(Number(process.env.TRACE_RATIO ?? "0.02")),
  }),
  spanProcessors: [processor],
});
sdk.start();

// ❌ AlwaysOnSampler in production at 50k RPS
import { AlwaysOnSampler } from "@opentelemetry/sdk-trace-base";

Force-keep errors at the edges

// src/http-hook.ts — after response, bump sampling bit for bad outcomes
import { trace, SpanStatusCode } from "@opentelemetry/api";

export function finalizeSpan(status: number, durationMs: number) {
  const span = trace.getActiveSpan();
  if (!span) return;
  if (status >= 500 || durationMs > 2000) {
    span.setStatus({ code: SpanStatusCode.ERROR });
    span.setAttribute("sampling.force", true);
    // Tail sampler / collector rule keys off this attribute or status
  }
}

Collector sketch (tail):

# otel-collector-config.yaml (excerpt)
processors:
  tail_sampling:
    decision_wait: 10s
    policies:
      - name: errors
        type: status_code
        status_code: { status_codes: [ERROR] }
      - name: slow
        type: latency
        latency: { threshold_ms: 2000 }
      - name: probabilistic
        type: probabilistic
        probabilistic: { sampling_percentage: 2 }

Budget math at 50k RPS

Closing checklist

✅ Dos
– ✅ Parent-based ratio for roots; respect upstream sample flags
– ✅ Tail-keep errors and slow requests
– ✅ Bound BatchSpanProcessor queues; monitor drops
– ✅ Separate trace pipeline SLOs from app SLOs
– ✅ Load-test collectors at peak, not average

❌ Don’ts
– ❌ Don’t run AlwaysOn in prod “just for a week”
– ❌ Don’t sample only in the app with no collector policy
– ❌ Don’t ignore exporter error/drop metrics
– ❌ Don’t attach huge payloads as span attributes

Related reading

• OpenTelemetry Tail Sampling: Full Fidelity on Errors, Sparse on Success
• OpenTelemetry for LLMs: Trace Prompt Latency Across Microservices
• Lambda X-Ray Sampling: Cost Versus Debuggability During Incidents
• Node Diagnostic Channels: Instrument Hot Paths Without Monkey Patches

Last updated on September 11, 2026