Node Flamegraphs on ECS: Continuous Profiling That Survives Scale

Node Flamegraphs on ECS: Continuous Profiling That Survives Scale
11 views
Always-on CPU profiles at 10k tasks will melt your observability bill and your privacy review. Always-off profiles mean you discover the hot function after the SEV call. The workable middle: continuous lightweight sampling with aggressive PII scrubbing, plus p99-triggered full captures that upload flamegraphs to S3 only when latency or CPU burn crosses a budget.
⚡ TL;DR: Prefer pprof-compatible sampling (--cpu-prof windows or clinic/0x/pyroscope-style agents) with 5–20 Hz defaults; gate heavy captures on CloudWatch/ADOT latency alarms; scrub route params and auth headers before upload; store artifacts per cluster/task-revision. Correlate with Node.js Event Loop Lag: Catch P99 Stalls and OpenTelemetry for LLMs: Trace Prompt Latency Across Microservices.

Node Flamegraphs on ECS: production guidance

Traces tell you which await was slow. Flamegraphs tell you which JS or native frames burned the core. On ECS, run a sidecar or in-process sampler that never blocks the event loop for more than a short safepoint.
typescript
// src/profiling/guarded-profile.ts
import { Session } from "node:inspector/promises";
import { writeFile } from "node:fs/promises";

export async function captureCpuProfile(seconds = 5): Promise<Buffer> {
 const session = new Session();
 session.connect();
 await session.post("Profiler.enable");
 await session.post("Profiler.start");
 await new Promise((r) => setTimeout(r, seconds * 1000));
 const { profile } = await session.post("Profiler.stop");
 session.disconnect();
 // ✅ Scrub before persist
 const scrubbed = scrubProfile(profile);
 const buf = Buffer.from(JSON.stringify(scrubbed));
 return buf;
}

function scrubProfile(profile: any) {
 // strip absolute paths → repo-relative; drop URL query strings in frame names
 for (const node of profile.nodes ?? []) {
 if (node.callFrame?.url) {
 node.callFrame.url = node.callFrame.url.replace(/\?.*$/, "").replace(/\/home\/[^/]+/, "/home/*");
 }
 }
 return profile;
}
typescript
// ❌ Unlimited --cpu-prof on every task forever, writing to container FS
// node --cpu-prof --cpu-prof-dir=/tmp app.js

Trigger on symptoms, not wall clocks alone

Wall-clock schedules create piles of idle profiles. Trigger on ECS service CPU > target, ALB p99, or event-loop delay histogram breaches.
typescript
// src/profiling/trigger.ts
export function shouldCapture(metricsSnapshot: {
 eventLoopP99Ms: number;
 cpuPct: number;
 alreadyCapturing: boolean;
}) {
 if (metricsSnapshot.alreadyCapturing) return false;
 return metricsSnapshot.eventLoopP99Ms > 50 || metricsSnapshot.cpuPct > 85;
}
yaml
# ecs task snippet — ephemeral volume for profiles
MountPoints:
 - SourceVolume: profiles
 ContainerPath: /var/profiles
 ReadOnly: false

Upload with tenancy and retention

Put objects under s3://corp-profiles/{env}/{service}/{taskDefRevision}/{taskId}/{ts}.cpuprofile with a 14-day lifecycle. Encrypt with a CMK; restrict GetObject to the platform debug role — same least-privilege instinct as Agent Tool Allowlists.
typescript
import { S3Client, PutObjectCommand } from "@aws-sdk/client-s3";

const s3 = new S3Client({});

export async function uploadProfile(key: string, body: Buffer) {
 await s3.send(new PutObjectCommand({
 Bucket: process.env.PROFILE_BUCKET!,
 Key: key,
 Body: body,
 ContentType: "application/json",
 ServerSideEncryption: "aws:kms",
 SSEKMSKeyId: process.env.PROFILE_KMS_KEY,
 }));
}

Privacy filters that pass legal review

  • Drop Authorization, Cookie, and x-api-key from any frame or label.
  • Hash user ids in custom span names before they leak into profile annotations.
  • Never attach request bodies to profiling context packs.
  • Gate production profiling behind a service feature flag and change ticket.

Closing checklist

✅ Dos
– ✅ Sample continuously light; capture heavy on p99/CPU triggers
– ✅ Scrub URLs and secrets before S3 upload
– ✅ Key artifacts by task definition revision
– ✅ Expire profiles automatically (7–14 days)
– ✅ Link profile timestamps to deploy and alarm ids ❌ Don’ts
– ❌ Don’t leave --cpu-prof on 100% of tasks indefinitely
– ❌ Don’t write profiles onto root FS without volume + rotation
– ❌ Don’t grant world-readable S3 ACLs on profiles
– ❌ Don’t profile staging-only and assume prod hot paths match
– ❌ Don’t ignore native frames from bcrypt/sharp/wasm

Related reading

Last updated on September 11, 2026 Related: Node.js Cluster Module: Scale Your API to All CPU Cores Without Kubernetes; Node.js Clustering and Worker Threads: Scale CPU-Bound Work Across All Cores; Graceful Shutdown for Node: Drain Correctly on Kubernetes and ECS.

Watch: 100 Days of AI on YouTube

Short videos from CheatCoders, one AI engineering topic per day.

Open the playlist on YouTube · CheatCoders on YouTube

Deep-dive PDF

Get the expanded guide for this post — extra diagrams-style checklists, failure modes, and a production walkthrough. Free when you subscribe to CheatCoders.

Already subscribed? or open the subscribe page.


Discover more from CheatCoders

Subscribe to get the latest posts sent to your email.

1 Comment

Leave a comment

No account needed. Name and email are optional.