Node Heap Snapshots in Production: Safe Capture Under Incident Load

Heap snapshots are the nuclear option for memory leaks — and nuclear options pause the world. Under incident load, an unthrottled v8.writeHeapSnapshot() can freeze the event loop for seconds, inflate RSS while serializing, and dump customer PII into an unencrypted tarball on an ephemeral disk. The unfair advantage is a gated capture path: rate-limited, redacted, streamed to S3, and never triggered by a panicked engineer SSH’ing into a hot pod.

⚡ TL;DR: Expose a signed admin endpoint (or SIGUSR2) that acquires a process-wide mutex, writes to /tmp with a hard size budget, scrubs string samples, uploads to a private S3 bucket with SSE-KMS, then deletes local files. Cap to 1 snapshot / 15 min / instance. Pair with Node Memory Leaks: Detached Cache Graphs and Node Flamegraphs on ECS.

Gate capture behind a mutex and budget

// src/heap-snapshot-gate.ts
import { writeHeapSnapshot } from "node:v8";
import { randomUUID } from "node:crypto";
import { unlinkSync, statSync, createReadStream } from "node:fs";
import { S3Client, PutObjectCommand } from "@aws-sdk/client-s3";

const s3 = new S3Client({});
let inflight = false;
let lastAt = 0;
const MIN_INTERVAL_MS = 15 * 60_000;
const MAX_BYTES = 2 * 1024 * 1024 * 1024;

export async function captureHeap(reason: string) {
  const now = Date.now();
  if (inflight) throw new Error("snapshot_inflight");
  if (now - lastAt < MIN_INTERVAL_MS) throw new Error("snapshot_rate_limited");
  inflight = true;
  lastAt = now;
  const id = randomUUID();
  const path = `/tmp/heap-${id}.heapsnapshot`;
  try {
    // ✅ Advertise pause to metrics before the stop-the-world
    metrics.gauge("heap.snapshot.paused", 1);
    writeHeapSnapshot(path);
    const size = statSync(path).size;
    if (size > MAX_BYTES) throw new Error(`snapshot_too_large:${size}`);
    await s3.send(new PutObjectCommand({
      Bucket: process.env.HEAP_BUCKET!,
      Key: `heaps/${process.env.SERVICE}/${id}.heapsnapshot`,
      Body: createReadStream(path),
      ServerSideEncryption: "aws:kms",
      Metadata: { reason },
    }));
    return { id, size };
  } finally {
    try { unlinkSync(path); } catch {}
    metrics.gauge("heap.snapshot.paused", 0);
    inflight = false;
  }
}

// ❌ Free-for-all SIGUSR2 with no rate limit during a sev-1
process.on("SIGUSR2", () => writeHeapSnapshot());

Scrub PII before anyone downloads

Chrome DevTools heap files contain string values. Treat them like prod dumps.

// src/scrub-heapsnapshot.ts
import { Transform } from "node:stream";

const SECRET = /(api[_-]?key|password|authorization|Bearer\s+\S+|eyJ[A-Za-z0-9_-]+\.[A-Za-z0-9_-]+)/gi;

export function scrubStrings() {
  return new Transform({
    transform(chunk, _enc, cb) {
      // ✅ Blank obvious secret-shaped strings in snapshot text
      cb(null, Buffer.from(String(chunk).replace(SECRET, "[REDACTED]")));
    },
  });
}

Never email raw .heapsnapshot files in Slack. Upload only to a bucket with object lock and short-lived signed URLs for the on-call who requested capture.

Capture under load without cascading

# ✅ Trigger on ONE ECS task, not the whole service
aws ecs execute-command --cluster prod --task $TASK \
  --interactive --command "node -e \"require('./dist/heap-snapshot-gate').captureHeap('sev1-rss')\""

Closing checklist

✅ Dos
– ✅ Rate-limit + single-flight every production snapshot
– ✅ Stream to SSE-KMS S3; delete local immediately
– ✅ Scrub secrets/PII-shaped strings before sharing
– ✅ Prefer one canary task over fleet-wide capture
– ✅ Record reason, size, and operator identity in metadata

❌ Don’ts
– ❌ Don’t bind raw writeHeapSnapshot to an unauthenticated HTTP route
– ❌ Don’t snapshot every pod during a memory incident
– ❌ Don’t leave multi-GB files on ephemeral disks overnight
– ❌ Don’t paste heap paths into public tickets

Related reading

• Node Memory Leaks: Detached Cache Graphs That Heap Snapshots Reveal
• Node Flamegraphs on ECS: Continuous Profiling That Survives Scale
• Event Loop Blockers: Perfetto Traces That Metrics Alone Will Miss
• AsyncLocalStorage Overhead: Measure Before Wrapping Every Request Path

Last updated on September 11, 2026