Node Core Dump Analysis: Symbolicate Native Crashes Inside Containers

Node Core Dump Analysis: Symbolicate Native Crashes Inside Containers

Native addons and sharp/libc bugs still SIGSEGV Node in production. Without cores you only see “Runtime exited.” The unfair advantage is controlled core capture on ECS: ulimit + pattern to EFS/S3, matching debug symbols, scrubbing, then lldb/gdb—never leaving dumps with customer payloads on shared disks.

⚡ TL;DR: Set ulimit -c and kernel.core_pattern (or ECS-compatible path); ship breakpad/async-profiler only when needed; symbolicate with the exact Node build + .node addon symbols; PII-scrub before ticket attach. Pair with N-API Addons for Node and Ephemeral Debug Sidecars.

Enable dumps without melting the disk

# ✅ Entry script (ECS)
ulimit -c unlimited
# Write to mounted volume, not overlay
echo "/cores/core.%e.%p.%t" > /tmp/core_pattern_hint
# Prefer host/sysctl managed by platform team:
# kernel.core_pattern = |/usr/lib/systemd/systemd-coredump  OR  /var/cores/core.%e.%p

export NODE_OPTIONS="--abort-on-uncaught-exception"  # optional; native faults dump regardless
# ✅ Keep a debug image with matching symbols (not in prod by default)
FROM public.ecr.aws/docker/library/node:20-bookworm
RUN apt-get update && apt-get install -y gdb lldb binutils \
  && mkdir -p /cores
# Copy exact addon .node + .debug files from CI artifacts

Symbolicate the crash

# ✅ gdb batch against core + node binary from the same build
gdb /usr/local/bin/node /cores/core.node.83.171000 \
  -batch \
  -ex "set pagination off" \
  -ex "bt" \
  -ex "thread apply all bt" \
  -ex "info sharedlibrary"
Artifact Why required
Exact node binary Frames won’t resolve otherwise
Addon .node + debug symbols Most segfaults are native
libc/libstdc++ from image Deep frames
Build-id / VCS SHA Prove match
// ❌ Shipping alpine prod + debugging with debian node — addresses never match

Scrub before share

Cores contain heap → tokens, PII, documents. Pipeline: encrypt at rest on the core volume → analyst pulls via audited sidecar → run strings allowlist / redact → attach only backtrace + module list to the ticket.

# ✅ Minimal shareable artifact
gdb ... -ex "bt" | tee stack.txt
# Do not upload raw core to Slack/Jira

Closing checklist

✅ Dos
– ✅ Platform-owned core pattern + retention (24–72h)
– ✅ CI publish debug symbols keyed by build-id
– ✅ Rate-limit dump capture (one per task family / hour)
– ✅ Use ephemeral debug sidecars for interactive gdb
– ✅ Reproduce under ASAN/UBSAN in staging for addon bugs

❌ Don’ts
– ❌ Don’t enable unlimited cores on tiny ephemeral disks
– ❌ Don’t paste core files into chat
– ❌ Don’t analyze with mismatched Node builds
– ❌ Don’t ignore addon version skew after “only JS changed” deploys

Related reading

Last updated on September 11, 2026


Discover more from CheatCoders

Subscribe to get the latest posts sent to your email.

Comments

No comments yet. Why don’t you start the discussion?

Leave a Reply