Node Flamegraphs on ECS: Continuous Profiling That Survives Scale

Always-on CPU profiles at 10k tasks will melt your observability bill and your privacy review. Always-off profiles mean you discover the hot function after the SEV call. The workable middle: continuous lightweight sampling with aggressive PII scrubbing, plus p99-triggered full captures that upload flamegraphs to S3 only when latency or CPU burn crosses a budget.

⚡ TL;DR: Prefer pprof-compatible sampling (--cpu-prof windows or clinic/0x/pyroscope-style agents) with 5–20 Hz defaults; gate heavy captures on CloudWatch/ADOT latency alarms; scrub route params and auth headers before upload; store artifacts per cluster/task-revision. Correlate with Node.js Event Loop Lag: Catch P99 Stalls and OpenTelemetry for LLMs: Trace Prompt Latency Across Microservices.

Sampling beats tracing for CPU attribution

Traces tell you which await was slow. Flamegraphs tell you which JS or native frames burned the core. On ECS, run a sidecar or in-process sampler that never blocks the event loop for more than a short safepoint.

// src/profiling/guarded-profile.ts
import { Session } from "node:inspector/promises";
import { writeFile } from "node:fs/promises";

export async function captureCpuProfile(seconds = 5): Promise<Buffer> {
  const session = new Session();
  session.connect();
  await session.post("Profiler.enable");
  await session.post("Profiler.start");
  await new Promise((r) => setTimeout(r, seconds * 1000));
  const { profile } = await session.post("Profiler.stop");
  session.disconnect();
  // ✅ Scrub before persist
  const scrubbed = scrubProfile(profile);
  const buf = Buffer.from(JSON.stringify(scrubbed));
  return buf;
}

function scrubProfile(profile: any) {
  // strip absolute paths → repo-relative; drop URL query strings in frame names
  for (const node of profile.nodes ?? []) {
    if (node.callFrame?.url) {
      node.callFrame.url = node.callFrame.url.replace(/\?.*$/, "").replace(/\/home\/[^/]+/, "/home/*");
    }
  }
  return profile;
}

// ❌ Unlimited --cpu-prof on every task forever, writing to container FS
// node --cpu-prof --cpu-prof-dir=/tmp app.js

Trigger on symptoms, not wall clocks alone

Wall-clock schedules create piles of idle profiles. Trigger on ECS service CPU > target, ALB p99, or event-loop delay histogram breaches.

// src/profiling/trigger.ts
export function shouldCapture(metricsSnapshot: {
  eventLoopP99Ms: number;
  cpuPct: number;
  alreadyCapturing: boolean;
}) {
  if (metricsSnapshot.alreadyCapturing) return false;
  return metricsSnapshot.eventLoopP99Ms > 50 || metricsSnapshot.cpuPct > 85;
}

# ecs task snippet — ephemeral volume for profiles
MountPoints:
  - SourceVolume: profiles
    ContainerPath: /var/profiles
    ReadOnly: false

Upload with tenancy and retention

Put objects under s3://corp-profiles/{env}/{service}/{taskDefRevision}/{taskId}/{ts}.cpuprofile with a 14-day lifecycle. Encrypt with a CMK; restrict GetObject to the platform debug role — same least-privilege instinct as Agent Tool Allowlists.

import { S3Client, PutObjectCommand } from "@aws-sdk/client-s3";

const s3 = new S3Client({});

export async function uploadProfile(key: string, body: Buffer) {
  await s3.send(new PutObjectCommand({
    Bucket: process.env.PROFILE_BUCKET!,
    Key: key,
    Body: body,
    ContentType: "application/json",
    ServerSideEncryption: "aws:kms",
    SSEKMSKeyId: process.env.PROFILE_KMS_KEY,
  }));
}

Privacy filters that pass legal review

• Drop Authorization, Cookie, and x-api-key from any frame or label.
• Hash user ids in custom span names before they leak into profile annotations.
• Never attach request bodies to profiling context packs.
• Gate production profiling behind a service feature flag and change ticket.

Closing checklist

✅ Dos
– ✅ Sample continuously light; capture heavy on p99/CPU triggers
– ✅ Scrub URLs and secrets before S3 upload
– ✅ Key artifacts by task definition revision
– ✅ Expire profiles automatically (7–14 days)
– ✅ Link profile timestamps to deploy and alarm ids

❌ Don’ts
– ❌ Don’t leave --cpu-prof on 100% of tasks indefinitely
– ❌ Don’t write profiles onto root FS without volume + rotation
– ❌ Don’t grant world-readable S3 ACLs on profiles
– ❌ Don’t profile staging-only and assume prod hot paths match
– ❌ Don’t ignore native frames from bcrypt/sharp/wasm

Related reading

• Node.js Event Loop Lag: Catch P99 Stalls
• OpenTelemetry for LLMs: Trace Prompt Latency Across Microservices
• Lambda Warm Pools: Low-Latency Backends for Coding Agent Tools
• CI Failure Triage Bots: Separate Flaky Noise from Real Regressions

Last updated on September 11, 2026