Python Deadlock Debugging: faulthandler Plus gdb for Wedged Workers

When a worker stops answering health checks but CPU is idle, you are usually in a lock graph—not a “slow query.” The unfair advantage is dumping all Python threads with faulthandler and, when that is silent, attaching gdb for native frames inside C extensions.

⚡ TL;DR: Enable faulthandler dump-on-signal and periodic dumps; on wedge send SIGUSR1/SIGABRT per policy; if stacks show native wait, use gdb + py-bt; fix lock ordering across Python and extension locks. Pair with Distributed Deadlock Detection and Python Prefork Pitfalls.

Always-on faulthandler

# ✅ boot path in gunicorn/uvicorn worker
import faulthandler
import signal
import sys

faulthandler.enable(file=sys.stderr, all_threads=True)
# Dump all threads on SIGUSR1 without killing the process
faulthandler.register(signal.SIGUSR1, file=sys.stderr, all_threads=True)
# Optional: periodic dump every 5 minutes while debugging a fleet
# faulthandler.dump_traceback_later(300, repeat=True, file=sys.stderr)

# From the host / sidecar when readiness fails
kill -USR1 $(pgrep -f "uvicorn.*worker")
# stderr now has every Python thread stack

When Python stacks lie

If every thread is in PyEval_RestoreThread / extension pthread_mutex_lock, you need native frames.

# ✅ gdb against the wedged PID (ECS exec / privileged debug sidecar)
gdb -p "$PID" -batch \
  -ex "set pagination off" \
  -ex "thread apply all bt" \
  -ex "thread apply all py-bt" \
  -ex "detach" -ex "quit"

# Reproduce classic extension + GIL hazard in staging
import threading
lock_a = threading.Lock()
lock_b = threading.Lock()

def t1():
    with lock_a:
        with lock_b:
            pass

def t2():
    with lock_b:
        with lock_a:  # ❌ inverted order → deadlock under race
            pass

Production playbook

1. Confirm wedge: readiness fail, no RPS, CPU ~0, FDs stable.
2. kill -USR1 → collect stacks to CloudWatch / file.
3. If stuck in native code → ephemeral debug sidecar with matching symbols.
4. Map lock graph; enforce global order; add timeouts (lock.acquire(timeout=…)).
5. Add a canary that holds inverted locks in CI so the bug fails before prod.

❌ Restarting the task as the only remediation — you destroy evidence and train the org to ignore systemic lock bugs.

Closing checklist

✅ Dos
– ✅ Register faulthandler on worker boot
– ✅ Document which signal dumps vs kills
– ✅ Keep debug images with CPython debug symbols available
– ✅ Enforce lock ordering docs in CODEOWNERS-owned modules
– ✅ Prefer timeouts on cross-component locks

❌ Don’ts
– ❌ Don’t ship without a dump signal in production
– ❌ Don’t attach gdb to every task by default (audit + ephemeral only)
– ❌ Don’t ignore wedges that “clear after bounce”
– ❌ Don’t mix multiprocessing locks with threads without a design

Related reading

• Distributed Deadlock Detection
• Python Prefork Pitfalls
• Node Heap Snapshots in Production
• Ephemeral Debug Sidecars

Last updated on September 11, 2026