Distributed Deadlock Detection: Cycles Across Postgres Rows and Redis Locks

Postgres detects row deadlocks inside one database. It cannot see that transaction A holds Redis key lock:order while waiting on row B, and transaction B holds row B while waiting on Redis. The unfair advantage is a global lock order plus timeouts across both systems, with traces that prove the cycle.

⚡ TL;DR: Define a total order (e.g. Redis → Postgres → external HTTP never while holding DB); set lock_timeout / Redis NX TTL; never take Redis while in an open transaction unless ordered; dump both sides on wedge. Pair with Postgres lock_timeout Strategies and Distributed Locks Reality Check.

The cross-system cycle

Tx1: SET NX redis:lock:42 → BEGIN → UPDATE accounts WHERE id=1  (waits)
Tx2: BEGIN → UPDATE accounts WHERE id=1 (holds) → SET NX redis:lock:42 (waits)
→ distributed deadlock; Postgres alone won't abort Tx1/Tx2 together

# ❌ Redis lock inside an open Postgres transaction
async with pool.connection() as conn:
    async with conn.transaction():
        await redis.set("lock:42", "1", nx=True, ex=30)
        await conn.execute("UPDATE accounts SET bal=bal-1 WHERE id=1")
        # partner takes row first then Redis → wedge

# ✅ Global order: Redis (if needed) BEFORE BEGIN, or use DB-only locks
ok = await redis.set("lock:42", owner, nx=True, ex=30)
if not ok:
    raise Busy()
try:
    async with pool.connection() as conn:
        async with conn.transaction():
            await conn.execute(
                "UPDATE accounts SET bal=bal-1 WHERE id=1"
            )
finally:
    if await redis.get("lock:42") == owner:
        await redis.delete("lock:42")

Timeouts as cycle breakers

-- ✅ session defaults for API workers
SET lock_timeout = '2s';
SET statement_timeout = '5s';
SET idle_in_transaction_session_timeout = '10s';

Detection playbook

1. App threads blocked on Redis + pg_locks / pg_stat_activity waiting.
2. Correlate by request_id in both Redis client logs and Postgres application_name.
3. Fix ordering; add tests that deliberately invert order and assert timeout (not hang).

Closing checklist

✅ Dos
– ✅ Publish a global lock-order ADR
– ✅ Set lock/statement timeouts on every pool
– ✅ Prefer single-system locks for a critical section
– ✅ Propagate request IDs into Redis + Postgres
– ✅ Chaos-test inverted order in CI

❌ Don’ts
– ❌ Don’t acquire Redis while holding Postgres rows unless order is proven safe
– ❌ Don’t use infinite blocking Redis BLPOP inside transactions
– ❌ Don’t rely on Postgres deadlock detector alone
– ❌ Don’t leave idle in transaction connections from ORMs

Related reading

• Postgres lock_timeout Strategies
• Distributed Locks Reality Check: When Redis Redlock Is the Wrong Tool
• Python Deadlock Debugging: faulthandler Plus gdb
• Exactly-Once Illusions

Last updated on September 11, 2026