Lambda Telemetry API Extensions: Custom Samplers Under Five Milliseconds

Lambda Telemetry API Extensions: Custom Samplers Under Five Milliseconds
8 views
Extensions that “just ship everything to our collector” add milliseconds you will not notice in dashboards until checkout p99 melts. The unfair advantage is a Telemetry API extension that samples under a hard 5ms budget per invoke: ring buffers, probabilistic keep, and drop-on-backpressure — never block the runtime.
⚡ TL;DR: Subscribe to platform + selective function streams. Keep extension work off the invoke hot path with async queues. Sample by trace flag, error bit, and tenant tier — not “100% always.” Fail open: if the collector stalls, drop spans, do not stall Lambda. Correlate with OpenTelemetry for LLMs and keep agent sandboxes honest via Secure AI Sandboxes.

Lambda Telemetry API Extensions: production guidance

typescript
// extension/index.ts — illustrative Telemetry API client
const LISTENER = `http://${process.env.AWS_LAMBDA_RUNTIME_API}/2022-07-01/telemetry`;

await fetch(LISTENER, {
 method: "PUT",
 body: JSON.stringify({
 schemaVersion: "2022-11-29",
 types: ["platform", "function"],
 buffering: { timeoutMs: 100, maxBytes: 256_000, maxItems: 200 },
 destination: { protocol: "HTTP", URI: "http://sandbox:4321" },
 }),
});

const queue: unknown[] = [];
const MAX_QUEUE = 500;

Deno.serve({ port: 4321 }, async (req) => {
 const batch = await req.json();
 for (const e of batch) {
 if (queue.length >= MAX_QUEUE) break; // DROP — never block
 if (shouldSample(e)) queue.push(e);
 }
 return new Response("ok");
});

function shouldSample(e: any): boolean {
 if (e.type === "platform.runtimeDone" && e.record?.status === "error") return true;
 if (e.record?.tracing?.sampled) return true;
 return Math.random() < 0.05; // 5% baseline
}
✅ Drop on full queue; error spans always kept.
❌ Synchronously POSTing each span to a remote collector inside the extension receive handler without timeouts.

Stay under five milliseconds

typescript
// Measure extension overhead with platform reports + custom metric
// Target: p99(extension_processing_ms) < 5 for latency-sensitive functions

export function assertBudget(started: number) {
 const dt = performance.now() - started;
 if (dt > 5) {
 // metric only — do not throw into the runtime
 process.stdout.write(JSON.stringify({ _aws: { Timestamp: Date.now(), CloudWatchMetrics: [{
 Namespace: "LambdaExt", MetricData: [{ MetricName: "SampleOverBudget", Value: 1, Unit: "Count" }],
 }]}}) + "
");
 }
}
Use buffering (timeoutMs/maxItems) so you amortize syscalls. Prefer local Unix/HTTP sandbox listeners over TLS to a distant backend on every batch.

Sampling policy that still debugs Sev-1s

Signal Sample rate
platform.* errors / timeouts 100%
Function spans with error=true 100%
High-tier tenants / canaries 50–100%
Baseline success path 1–5%
During declared incident window temporary 100% via env/SSM
Wire incident boosts carefully so you do not permanently 10× the bill — same discipline as X-Ray incident sampling.

Closing checklist

  • [ ] Telemetry subscription uses buffering; listener never blocks runtime
  • [ ] Queue has a max size; overflow drops (fail open)
  • [ ] Errors and pre-sampled traces always retained
  • [ ] p99 extension processing budget < 5ms alarmed
  • [ ] Collector export async with hard timeouts
  • [ ] Incident sample boost is time-boxed

Related reading

Last updated on September 11, 2026 Related: Lambda Extensions API: Build Custom Monitoring Without Touching Application Code; Lambda Function URLs with Streaming: Replace API Gateway for 90% of Use Cases; How to Build a Hyper-Scale Serverless API Platform on AWS with API Gateway, Lambda, DynamoDB, and CloudFront.

Watch: 100 Days of AI on YouTube

Short videos from CheatCoders, one AI engineering topic per day.

Open the playlist on YouTube · CheatCoders on YouTube

Deep-dive PDF

Get the expanded guide for this post — extra diagrams-style checklists, failure modes, and a production walkthrough. Free when you subscribe to CheatCoders.

Already subscribed? or open the subscribe page.


Discover more from CheatCoders

Subscribe to get the latest posts sent to your email.

Comments

No comments yet. Why don’t you start the discussion?

Leave a comment

No account needed. Name and email are optional.