11 views
Always-on CPU profiles at 10k tasks will melt your observability bill and your privacy review. Always-off profiles mean you discover the hot function after the SEV call. The workable middle: continuous lightweight sampling with aggressive PII scrubbing, plus p99-triggered full captures that upload flamegraphs to S3 only when latency or CPU burn crosses a budget.
⚡ TL;DR: Prefer pprof-compatible sampling (--cpu-profwindows orclinic/0x/pyroscope-style agents) with 5–20 Hz defaults; gate heavy captures on CloudWatch/ADOT latency alarms; scrub route params and auth headers before upload; store artifacts per cluster/task-revision. Correlate with Node.js Event Loop Lag: Catch P99 Stalls and OpenTelemetry for LLMs: Trace Prompt Latency Across Microservices.
Node Flamegraphs on ECS: production guidance
Traces tell you which await was slow. Flamegraphs tell you which JS or native frames burned the core. On ECS, run a sidecar or in-process sampler that never blocks the event loop for more than a short safepoint.typescript
// src/profiling/guarded-profile.ts
import { Session } from "node:inspector/promises";
import { writeFile } from "node:fs/promises";
export async function captureCpuProfile(seconds = 5): Promise<Buffer> {
const session = new Session();
session.connect();
await session.post("Profiler.enable");
await session.post("Profiler.start");
await new Promise((r) => setTimeout(r, seconds * 1000));
const { profile } = await session.post("Profiler.stop");
session.disconnect();
// ✅ Scrub before persist
const scrubbed = scrubProfile(profile);
const buf = Buffer.from(JSON.stringify(scrubbed));
return buf;
}
function scrubProfile(profile: any) {
// strip absolute paths → repo-relative; drop URL query strings in frame names
for (const node of profile.nodes ?? []) {
if (node.callFrame?.url) {
node.callFrame.url = node.callFrame.url.replace(/\?.*$/, "").replace(/\/home\/[^/]+/, "/home/*");
}
}
return profile;
}
typescript
// ❌ Unlimited --cpu-prof on every task forever, writing to container FS
// node --cpu-prof --cpu-prof-dir=/tmp app.js
Trigger on symptoms, not wall clocks alone
Wall-clock schedules create piles of idle profiles. Trigger on ECS service CPU > target, ALB p99, or event-loop delay histogram breaches.typescript
// src/profiling/trigger.ts
export function shouldCapture(metricsSnapshot: {
eventLoopP99Ms: number;
cpuPct: number;
alreadyCapturing: boolean;
}) {
if (metricsSnapshot.alreadyCapturing) return false;
return metricsSnapshot.eventLoopP99Ms > 50 || metricsSnapshot.cpuPct > 85;
}
yaml
# ecs task snippet — ephemeral volume for profiles
MountPoints:
- SourceVolume: profiles
ContainerPath: /var/profiles
ReadOnly: false
Upload with tenancy and retention
Put objects unders3://corp-profiles/{env}/{service}/{taskDefRevision}/{taskId}/{ts}.cpuprofile with a 14-day lifecycle. Encrypt with a CMK; restrict GetObject to the platform debug role — same least-privilege instinct as Agent Tool Allowlists.
typescript
import { S3Client, PutObjectCommand } from "@aws-sdk/client-s3";
const s3 = new S3Client({});
export async function uploadProfile(key: string, body: Buffer) {
await s3.send(new PutObjectCommand({
Bucket: process.env.PROFILE_BUCKET!,
Key: key,
Body: body,
ContentType: "application/json",
ServerSideEncryption: "aws:kms",
SSEKMSKeyId: process.env.PROFILE_KMS_KEY,
}));
}
Privacy filters that pass legal review
- Drop Authorization, Cookie, and
x-api-keyfrom any frame or label. - Hash user ids in custom span names before they leak into profile annotations.
- Never attach request bodies to profiling context packs.
- Gate production profiling behind a service feature flag and change ticket.
Closing checklist
✅ Dos– ✅ Sample continuously light; capture heavy on p99/CPU triggers
– ✅ Scrub URLs and secrets before S3 upload
– ✅ Key artifacts by task definition revision
– ✅ Expire profiles automatically (7–14 days)
– ✅ Link profile timestamps to deploy and alarm ids ❌ Don’ts
– ❌ Don’t leave
--cpu-prof on 100% of tasks indefinitely– ❌ Don’t write profiles onto root FS without volume + rotation
– ❌ Don’t grant world-readable S3 ACLs on profiles
– ❌ Don’t profile staging-only and assume prod hot paths match
– ❌ Don’t ignore native frames from bcrypt/sharp/wasm
Related reading
- Node.js Event Loop Lag: Catch P99 Stalls
- OpenTelemetry for LLMs: Trace Prompt Latency Across Microservices
- Lambda Warm Pools: Low-Latency Backends for Coding Agent Tools
- CI Failure Triage Bots: Separate Flaky Noise from Real Regressions
Watch: 100 Days of AI on YouTube
Short videos from CheatCoders, one AI engineering topic per day.
Most viewed
- Python Decorators Explained: From Simple Wrappers to Production Patterns
- Claude Code vs Cursor vs GitHub Copilot: Choose by Workflow, and Set Up One Repo That Works With All Three
- AI Agent Frameworks in 2025: LangGraph vs CrewAI vs AutoGen vs Raw API
- Python String Methods: Every str Method With Real Production Examples
- Css Animation Performance: Performance-Optimized Motion for Modern UIs
Newly added
- AWS Systems Manager Parameter Store: Hierarchical Runtime Config for Coding Agents
- Amazon Cognito Identity Pools: Federate Coding-Agent Tool Callers Without Long-Lived Keys
- Amazon Athena for Coding Agents: Query Tool Audit Logs and Eval Traces on S3 Without a Warehouse
- AWS Batch for Coding Agents: Parallel Overnight PR Reviews and Long-Running Jobs Without Lambda Timeouts
- Amazon Bedrock Prompt Management: Version and A/B Test Coding-Agent System Prompts
Deep-dive PDF
Get the expanded guide for this post — extra diagrams-style checklists, failure modes, and a production walkthrough. Free when you subscribe to CheatCoders.
Already subscribed? or open the subscribe page.
Discover more from CheatCoders
Subscribe to get the latest posts sent to your email.

Pingback: Node Heap Snapshots in Production: Safe Capture Under Incident Load - CheatCoders