10 views
Postgres detects row deadlocks inside one database. It cannot see that transaction A holds Redis key lock:order while waiting on row B, and transaction B holds row B while waiting on Redis. The unfair advantage is a global lock order plus timeouts across both systems, with traces that prove the cycle.
⚡ TL;DR: Define a total order (e.g. Redis → Postgres → external HTTP never while holding DB); set lock_timeout / Redis NX TTL; never take Redis while in an open transaction unless ordered; dump both sides on wedge. Pair with Postgres lock_timeout Strategies and Distributed Locks Reality Check.
Distributed Deadlock Detection: production guidance
text
Tx1: SET NX redis:lock:42 → BEGIN → UPDATE accounts WHERE id=1 (waits)
Tx2: BEGIN → UPDATE accounts WHERE id=1 (holds) → SET NX redis:lock:42 (waits)
→ distributed deadlock; Postgres alone won't abort Tx1/Tx2 together
python
# ❌ Redis lock inside an open Postgres transaction
async with pool.connection() as conn:
async with conn.transaction():
await redis.set("lock:42", "1", nx=True, ex=30)
await conn.execute("UPDATE accounts SET bal=bal-1 WHERE id=1")
# partner takes row first then Redis → wedge
python
# ✅ Global order: Redis (if needed) BEFORE BEGIN, or use DB-only locks
ok = await redis.set("lock:42", owner, nx=True, ex=30)
if not ok:
raise Busy()
try:
async with pool.connection() as conn:
async with conn.transaction():
await conn.execute(
"UPDATE accounts SET bal=bal-1 WHERE id=1"
)
finally:
if await redis.get("lock:42") == owner:
await redis.delete("lock:42")
Timeouts as cycle breakers
| Layer | Setting | Behavior |
|---|---|---|
| Postgres | lock_timeout=2s |
Fail fast waiters |
| Postgres | idle_in_transaction_session_timeout |
Kill abandoned txs |
| Redis | short TTL + fence | Lock expires |
| App | overall deadline | Cancel and release |
sql
-- ✅ session defaults for API workers
SET lock_timeout = '2s';
SET statement_timeout = '5s';
SET idle_in_transaction_session_timeout = '10s';
Detection playbook
- App threads blocked on Redis +
pg_locks/pg_stat_activitywaiting. - Correlate by
request_idin both Redis client logs and Postgresapplication_name. - Fix ordering; add tests that deliberately invert order and assert timeout (not hang).
Closing checklist
✅ Dos – ✅ Publish a global lock-order ADR – ✅ Set lock/statement timeouts on every pool – ✅ Prefer single-system locks for a critical section – ✅ Propagate request IDs into Redis + Postgres – ✅ Chaos-test inverted order in CI ❌ Don’ts – ❌ Don’t acquire Redis while holding Postgres rows unless order is proven safe – ❌ Don’t use infinite blocking RedisBLPOP inside transactions
– ❌ Don’t rely on Postgres deadlock detector alone
– ❌ Don’t leave idle in transaction connections from ORMs
Related reading
- Postgres lock_timeout Strategies
- Distributed Locks Reality Check: When Redis Redlock Is the Wrong Tool
- Python Deadlock Debugging: faulthandler Plus gdb
- Exactly-Once Illusions
Watch: 100 Days of AI on YouTube
Short videos from CheatCoders, one AI engineering topic per day.
Most viewed
- Python Decorators Explained: From Simple Wrappers to Production Patterns
- Claude Code vs Cursor vs GitHub Copilot: Choose by Workflow, and Set Up One Repo That Works With All Three
- AI Agent Frameworks in 2025: LangGraph vs CrewAI vs AutoGen vs Raw API
- Python String Methods: Every str Method With Real Production Examples
- Css Animation Performance: Performance-Optimized Motion for Modern UIs
Newly added
- AWS Systems Manager Parameter Store: Hierarchical Runtime Config for Coding Agents
- Amazon Cognito Identity Pools: Federate Coding-Agent Tool Callers Without Long-Lived Keys
- Amazon Athena for Coding Agents: Query Tool Audit Logs and Eval Traces on S3 Without a Warehouse
- AWS Batch for Coding Agents: Parallel Overnight PR Reviews and Long-Running Jobs Without Lambda Timeouts
- Amazon Bedrock Prompt Management: Version and A/B Test Coding-Agent System Prompts
Deep-dive PDF
Get the expanded guide for this post — extra diagrams-style checklists, failure modes, and a production walkthrough. Free when you subscribe to CheatCoders.
Already subscribed? or open the subscribe page.
Discover more from CheatCoders
Subscribe to get the latest posts sent to your email.
