Event Loop Blockers: Perfetto Traces That Metrics Alone Will Miss

Event Loop Blockers: Perfetto Traces That Metrics Alone Will Miss

perf_hooks.monitorEventLoopDelay tells you something blocked. It will not tell you that JSON.parse on a 12MB webhook or a sync pbkdf2 in a rare branch is the culprit. Perfetto / Chrome DevTools timelines (and Node’s trace events) give you wall-clock stacks across JS and native frames so you can see the stall, not just the symptom histogram.

⚡ TL;DR: Keep ELD histograms in prod; in staging (and during SEVs) capture Chrome-format CPU + timeline traces around the failing request; hunt long tasks >16–50ms on the main thread; fix by moving CPU to workers or streaming parsers. Companion pieces: Node.js Event Loop Lag: Catch P99 Stalls and Node Flamegraphs on ECS.

Metrics find smoke; traces find the fire

// src/eld.ts — always-on cheap signal
import { monitorEventLoopDelay } from "node:perf_hooks";

const h = monitorEventLoopDelay({ resolution: 10 });
h.enable();

setInterval(() => {
  metrics.gauge("node.eld_p99_ms", h.percentile(99) / 1e6);
  metrics.gauge("node.eld_max_ms", h.max / 1e6);
  h.reset();
}, 10_000);

When p99 ELD spikes without matching upstream latency, you need a timeline.

# ✅ Staging capture (Chrome DevTools protocol via inspector)
node --inspect=0.0.0.0:9229 dist/server.js
# Then: chrome://inspect → Performance → record during repro
# Or export trace events:
node --trace-event-categories=v8,node,node.async_hooks dist/server.js

Classic blockers that metrics mis-attribute

// ❌ Sync parse on giant payloads in the request path
app.post("/hook", (req, res) => {
  const body = JSON.parse(req.rawBody.toString("utf8")); // 12MB stalls
  res.end("ok");
});

// ✅ Streaming / size-capped parse
import { once } from "node:events";
app.post("/hook", async (req, res) => {
  const chunks: Buffer[] = [];
  let n = 0;
  for await (const c of req) {
    n += c.length;
    if (n > 1_000_000) return res.status(413).end();
    chunks.push(c);
  }
  const body = JSON.parse(Buffer.concat(chunks).toString("utf8"));
  res.end("ok");
});
// ❌ crypto.pbkdf2Sync in login
const hash = crypto.pbkdf2Sync(password, salt, 310000, 32, "sha256");

// ✅ async API — returns to the loop between libuv turns
const hash = await pbkdf2(password, salt, 310000, 32, "sha256");

Annotate traces with user marks

import { performance } from "node:perf_hooks";

export function withSpan<T>(name: string, fn: () => T): T {
  performance.mark(`${name}:start`);
  try {
    return fn();
  } finally {
    performance.mark(`${name}:end`);
    performance.measure(name, `${name}:start`, `${name}:end`);
  }
}

Open the trace in ui.perfetto.dev / Chrome Performance — long yellow tasks that sit on the main thread between http.incoming and http.outgoing are your backlog.

Staging workflow that sticks

  1. Reproduce with a captured payload.
  2. Record 5–15s of Performance timeline + simultaneous ELD gauges.
  3. Sort long tasks; identify JS vs native (zlib, bcrypt, sharp).
  4. Move native CPU to worker_threads or async APIs; stream JSON.
  5. Re-measure ELD p99 under the same load profile.

For property-style concurrency checks after the fix, borrow ideas from AI Test Generation: Property Tests That Catch Race Conditions.

Closing checklist

✅ Dos
– ✅ Keep ELD histograms in production
– ✅ Capture Perfetto/Chrome traces in staging on ELD regressions
– ✅ Cap sync parse sizes; prefer async crypto
– ✅ Offload heavy CPU to worker threads
– ✅ Attach deploy id to trace artifacts

❌ Don’ts
– ❌ Don’t rely on average latency alone to find blockers
– ❌ Don’t *Sync crypto/fs on request paths
– ❌ Don’t JSON.parse unbounded webhook bodies
– ❌ Don’t leave --inspect open on public interfaces in prod
– ❌ Don’t confuse GC pauses with your own sync JS without traces

Related reading

Last updated on September 11, 2026


Discover more from CheatCoders

Subscribe to get the latest posts sent to your email.

1 Comment

Leave a Reply