Python tracemalloc in Workers: Find Retained Caches Without Noise

RSS charts tell you memory grew; they do not tell you which dict retained per-request objects again. tracemalloc can — if you sample carefully in long-running ECS workers. Turn it on continuously at full fidelity and you will DDoS yourself with allocation noise. The senior pattern is: low-overhead baseline, on-demand snapshots during suspected leaks, differential reports that highlight retained frames, and PII-safe artifact upload.

⚡ TL;DR: Start tracemalloc with few frames; take pairwise snapshots around suspect intervals; diff top allocators; rate-limit capture. Scrub dumps before S3. Prefer this for retained caches; use heap dumps sparingly. Analogous discipline: Node heap snapshots in production.

Start small, snapshot on signal

# worker/memprobe.py
import tracemalloc, os, time
from pathlib import Path

def start_probe():
    n = int(os.getenv("TRACEMALLOC_FRAMES", "8"))  # ✅ small default
    if not tracemalloc.is_tracing():
        tracemalloc.start(n)

def snapshot(label: str):
    start_probe()
    snap = tracemalloc.take_snapshot()
    snap = snap.filter_traces((
        tracemalloc.Filter(False, "<frozen importlib._bootstrap>"),
        tracemalloc.Filter(False, "*site-packages/pip/*"),
    ))
    path = Path(f"/tmp/tracemalloc-{label}-{int(time.time())}.out")
    top = snap.statistics("lineno")[:30]
    path.write_text("\n".join(str(s) for s in top), encoding="utf-8")
    return path

Differential reports beat absolute lists

def diff_snapshots(before, after, limit=20):
    stats = after.compare_to(before, "lineno")
    # ✅ Focus on increases — ignore steady allocator chatter
    growth = [s for s in stats if s.size_diff > 0][:limit]
    return growth

# Usage in worker heartbeat when RSS delta > threshold
# before = tracemalloc.take_snapshot()
# ... process N jobs ...
# after = tracemalloc.take_snapshot()
# report(diff_snapshots(before, after))

❌ Printing top allocators once at idle — you will “fix” the JSON library forever and miss the cache that only grows on traffic.

Rate limits and privacy

# ✅ Gate on RSS slope, not every message
if rss_mb - last_rss_mb > 64 and time.time() - last_capture > 300:
    run_diff_capture()
    last_capture = time.time()

Look specifically for unbounded module-level dicts, lru_cache without bounds, and memoization keyed by raw request payloads. Fix with maxsize, WeakValue dictionaries, or request-scoped caches that die with the context.

Closing checklist

✅ Dos
– ✅ Use few frames and filtered diffs
– ✅ Trigger on RSS growth alarms, not every job
– ✅ Compare snapshot pairs over a work interval
– ✅ Scrub and TTL snapshot artifacts in S3
– ✅ Hunt unbounded caches and memo maps first

❌ Don’ts
– ❌ Don’t leave full-fidelity tracemalloc on 24/7 in huge fleets
– ❌ Don’t upload raw snapshots with customer payloads
– ❌ Don’t trust a single idle top-N list
– ❌ Don’t ignore lru_cache without maxsize
– ❌ Don’t capture many workers simultaneously during an incident

Related reading

• Node Heap Snapshots in Production: Safe Capture Under Incident Load
• Node Memory Leaks: Detached Cache Graphs That Heap Snapshots Reveal
• Python GIL Contention: Logging Handlers That Stall Hot Request Paths
• Event Loop Blockers: Perfetto Traces That Metrics Alone Will Miss

Last updated on September 11, 2026