Lambda Telemetry API Extensions: Custom Samplers Under Five Milliseconds

Extensions that “just ship everything to our collector” add milliseconds you will not notice in dashboards until checkout p99 melts. The unfair advantage is a Telemetry API extension that samples under a hard 5ms budget per invoke: ring buffers, probabilistic keep, and drop-on-backpressure — never block the runtime.

⚡ TL;DR: Subscribe to platform + selective function streams. Keep extension work off the invoke hot path with async queues. Sample by trace flag, error bit, and tenant tier — not “100% always.” Fail open: if the collector stalls, drop spans, do not stall Lambda. Correlate with OpenTelemetry for LLMs and keep agent sandboxes honest via Secure AI Sandboxes.

Budget-first extension loop

// extension/index.ts — illustrative Telemetry API client
const LISTENER = `http://${process.env.AWS_LAMBDA_RUNTIME_API}/2022-07-01/telemetry`;

await fetch(LISTENER, {
  method: "PUT",
  body: JSON.stringify({
    schemaVersion: "2022-11-29",
    types: ["platform", "function"],
    buffering: { timeoutMs: 100, maxBytes: 256_000, maxItems: 200 },
    destination: { protocol: "HTTP", URI: "http://sandbox:4321" },
  }),
});

const queue: unknown[] = [];
const MAX_QUEUE = 500;

Deno.serve({ port: 4321 }, async (req) => {
  const batch = await req.json();
  for (const e of batch) {
    if (queue.length >= MAX_QUEUE) break; // DROP — never block
    if (shouldSample(e)) queue.push(e);
  }
  return new Response("ok");
});

function shouldSample(e: any): boolean {
  if (e.type === "platform.runtimeDone" && e.record?.status === "error") return true;
  if (e.record?.tracing?.sampled) return true;
  return Math.random() < 0.05; // 5% baseline
}

✅ Drop on full queue; error spans always kept.
❌ Synchronously POSTing each span to a remote collector inside the extension receive handler without timeouts.

Stay under five milliseconds

// Measure extension overhead with platform reports + custom metric
// Target: p99(extension_processing_ms) < 5 for latency-sensitive functions

export function assertBudget(started: number) {
  const dt = performance.now() - started;
  if (dt > 5) {
    // metric only — do not throw into the runtime
    process.stdout.write(JSON.stringify({ _aws: { Timestamp: Date.now(), CloudWatchMetrics: [{
      Namespace: "LambdaExt", MetricData: [{ MetricName: "SampleOverBudget", Value: 1, Unit: "Count" }],
    }]}}) + "
");
  }
}

Use buffering (timeoutMs/maxItems) so you amortize syscalls. Prefer local Unix/HTTP sandbox listeners over TLS to a distant backend on every batch.

Sampling policy that still debugs Sev-1s

Wire incident boosts carefully so you do not permanently 10× the bill — same discipline as X-Ray incident sampling.

Closing checklist

• [ ] Telemetry subscription uses buffering; listener never blocks runtime
• [ ] Queue has a max size; overflow drops (fail open)
• [ ] Errors and pre-sampled traces always retained
• [ ] p99 extension processing budget < 5ms alarmed
• [ ] Collector export async with hard timeouts
• [ ] Incident sample boost is time-boxed

Related reading

• OpenTelemetry for LLMs: Trace Prompt Latency Across Microservices
• Lambda Warm Pools: Low-Latency Backends for Coding Agent Tools
• Secure AI Sandboxes: Ephemeral ECS Tasks for Agent Tool Execution
• CI Failure Triage Bots: Separate Flaky Noise From Real Regressions

Last updated on September 11, 2026