12 views
RSS charts tell you memory grew; they do not tell you which dict retained per-request objects again. tracemalloc can — if you sample carefully in long-running ECS workers. Turn it on continuously at full fidelity and you will DDoS yourself with allocation noise. The senior pattern is: low-overhead baseline, on-demand snapshots during suspected leaks, differential reports that highlight retained frames, and PII-safe artifact upload.
⚡ TL;DR: Start tracemalloc with few frames; take pairwise snapshots around suspect intervals; diff top allocators; rate-limit capture. Scrub dumps before S3. Prefer this for retained caches; use heap dumps sparingly. Analogous discipline: Node heap snapshots in production.
Python tracemalloc in Workers: production guidance
python
# worker/memprobe.py
import tracemalloc, os, time
from pathlib import Path
def start_probe():
n = int(os.getenv("TRACEMALLOC_FRAMES", "8")) # ✅ small default
if not tracemalloc.is_tracing():
tracemalloc.start(n)
def snapshot(label: str):
start_probe()
snap = tracemalloc.take_snapshot()
snap = snap.filter_traces((
tracemalloc.Filter(False, "<frozen importlib._bootstrap>"),
tracemalloc.Filter(False, "*site-packages/pip/*"),
))
path = Path(f"/tmp/tracemalloc-{label}-{int(time.time())}.out")
top = snap.statistics("lineno")[:30]
path.write_text("\n".join(str(s) for s in top), encoding="utf-8")
return path
Differential reports beat absolute lists
python
❌ Printing top allocators once at idle — you will “fix” the JSON library forever and miss the cache that only grows on traffic.
def diff_snapshots(before, after, limit=20):
stats = after.compare_to(before, "lineno")
# ✅ Focus on increases — ignore steady allocator chatter
growth = [s for s in stats if s.size_diff > 0][:limit]
return growth
# Usage in worker heartbeat when RSS delta > threshold
# before = tracemalloc.take_snapshot()
# ... process N jobs ...
# after = tracemalloc.take_snapshot()
# report(diff_snapshots(before, after))
Rate limits and privacy
| Control | Default |
|---|---|
| Frames | 5–10 |
| Snapshot pair interval | On RSS slope alarm, not every request |
| Concurrent captures | 1 per task |
| Artifact destination | Scrubbed S3 prefix, CMK, short TTL |
| Filter | Drop importlib / known noisy sites |
python
Look specifically for unbounded module-level dicts, # ✅ Gate on RSS slope, not every message
if rss_mb - last_rss_mb > 64 and time.time() - last_capture > 300:
run_diff_capture()
last_capture = time.time()
lru_cache without bounds, and memoization keyed by raw request payloads. Fix with maxsize, WeakValue dictionaries, or request-scoped caches that die with the context.
Closing checklist
✅ Dos – ✅ Use few frames and filtered diffs – ✅ Trigger on RSS growth alarms, not every job – ✅ Compare snapshot pairs over a work interval – ✅ Scrub and TTL snapshot artifacts in S3 – ✅ Hunt unbounded caches and memo maps first ❌ Don’ts – ❌ Don’t leave full-fidelity tracemalloc on 24/7 in huge fleets – ❌ Don’t upload raw snapshots with customer payloads – ❌ Don’t trust a single idle top-N list – ❌ Don’t ignorelru_cache without maxsize
– ❌ Don’t capture many workers simultaneously during an incident
Related reading
- Node Heap Snapshots in Production: Safe Capture Under Incident Load
- Node Memory Leaks: Detached Cache Graphs That Heap Snapshots Reveal
- Python GIL Contention: Logging Handlers That Stall Hot Request Paths
- Event Loop Blockers: Perfetto Traces That Metrics Alone Will Miss
Watch: 100 Days of AI on YouTube
Short videos from CheatCoders, one AI engineering topic per day.
Most viewed
- Python Decorators Explained: From Simple Wrappers to Production Patterns
- Claude Code vs Cursor vs GitHub Copilot: Choose by Workflow, and Set Up One Repo That Works With All Three
- AI Agent Frameworks in 2025: LangGraph vs CrewAI vs AutoGen vs Raw API
- Python String Methods: Every str Method With Real Production Examples
- Css Animation Performance: Performance-Optimized Motion for Modern UIs
Newly added
- AWS Systems Manager Parameter Store: Hierarchical Runtime Config for Coding Agents
- Amazon Cognito Identity Pools: Federate Coding-Agent Tool Callers Without Long-Lived Keys
- Amazon Athena for Coding Agents: Query Tool Audit Logs and Eval Traces on S3 Without a Warehouse
- AWS Batch for Coding Agents: Parallel Overnight PR Reviews and Long-Running Jobs Without Lambda Timeouts
- Amazon Bedrock Prompt Management: Version and A/B Test Coding-Agent System Prompts
Deep-dive PDF
Get the expanded guide for this post — extra diagrams-style checklists, failure modes, and a production walkthrough. Free when you subscribe to CheatCoders.
Already subscribed? or open the subscribe page.
Discover more from CheatCoders
Subscribe to get the latest posts sent to your email.
