28 views
A single long ALTER TABLE or forgotten SELECT … FOR UPDATE can stack hundreds of waiters. Apps retry, connection pools fill, ALB targets flip unhealthy, and you get a “database is down” page that was really a lock queue. Fail-fast timeouts turn that cascade into bounded, actionable errors.
⚡ TL;DR: Setlock_timeout(and usuallystatement_timeout) per role/session for OLTP traffic; keep migrations on a separate role with explicit longer budgets. Monitorpg_locks/pg_stat_activitywait events. Prefer skip-locked and short transactions. Pair with Distributed Deadlock Detection, Python Pool Timeouts, and SLA Error Budgets.
Postgres lock timeout Strategies: production guidance
text
Long holder (migration / idle-in-tx)
-> waiter 1..N block on relation/row lock
-> app pool exhausts
-> upstream timeouts + retries amplify load
-> "everything 500" without a single crash
lock_timeout aborts the waiter after N ms instead of joining an unbounded queue. The holder still needs fixing—but the blast radius shrinks.
Role-scoped defaults
sql
-- OLTP app role: fail fast on locks and statements
ALTER ROLE app_oltp SET lock_timeout = '2s';
ALTER ROLE app_oltp SET statement_timeout = '15s';
ALTER ROLE app_oltp SET idle_in_transaction_session_timeout = '30s';
-- Migration role: longer lock budget, still bounded
ALTER ROLE app_migrate SET lock_timeout = '5s';
ALTER ROLE app_migrate SET statement_timeout = '10min';
typescript
✅ Separate migration role with explicit budgets.// GOOD: set on checkout for Node (pg)
pool.on("connect", (client) => {
client.query("SET lock_timeout = '2s'");
client.query("SET statement_timeout = '15s'");
});
// BAD: rely on cluster defaults of 0 (wait forever)
❌ One superuser role for app traffic and DDL.
Application patterns that respect timeouts
sql
-- Prefer non-blocking claim patterns for workers
UPDATE jobs
SET status = 'running', locked_at = now()
WHERE id = (
SELECT id FROM jobs
WHERE status = 'queued'
ORDER BY id
FOR UPDATE SKIP LOCKED
LIMIT 1
)
RETURNING *;
typescript
Map try {
await db.query("UPDATE accounts SET balance = balance - $1 WHERE id = $2", [amt, id]);
} catch (e: any) {
if (e.code === "55P03") { // lock_not_available
metrics.increment("pg.lock_timeout");
throw new TransientError("lock_timeout"); // retry with jitter, or shed
}
throw e;
}
55P03 and 57014 (query_canceled) to retry/shed policies that match your SLA error budgets. Do not blindly retry forever—that recreates the queue.
Observability during incidents
sql
Alert when lock waiters exceed N for M minutes, or when SELECT pid, usename, state, wait_event_type, wait_event,
now() - query_start AS runtime, left(query, 120)
FROM pg_stat_activity
WHERE wait_event_type = 'Lock'
ORDER BY runtime DESC;
lock_timeout error rate spikes. Tie to Distributed Deadlock Detection when Redis fencing and Postgres rows interact.
Closing checklist
- [ ] OLTP roles have non-zero
lock_timeoutandstatement_timeout - [ ] Migrations use a separate role with documented budgets
- [ ] App maps
55P03to transient/shed paths, not infinite retries - [ ] Workers prefer
SKIP LOCKEDover blocking claims - [ ] Idle-in-transaction timeout enabled
- [ ] Alerts on lock waiter depth and timeout error rate
Related reading
- Distributed Deadlock Detection: Cycles Across Postgres Rows and Redis Locks
- Python Pool Timeouts: Connection Checkout Bugs That Look Like 500s
- SLA Error Budgets: Drive Automated Load Shedding Before Hard Outages
- Exactly-Once Illusions: Design At-Least-Once Plus Truly Idempotent Handlers
Watch: 100 Days of AI on YouTube
Short videos from CheatCoders, one AI engineering topic per day.
Most viewed
- Python Decorators Explained: From Simple Wrappers to Production Patterns
- Claude Code vs Cursor vs GitHub Copilot: Choose by Workflow, and Set Up One Repo That Works With All Three
- AI Agent Frameworks in 2025: LangGraph vs CrewAI vs AutoGen vs Raw API
- Python String Methods: Every str Method With Real Production Examples
- Css Animation Performance: Performance-Optimized Motion for Modern UIs
Newly added
- AWS Systems Manager Parameter Store: Hierarchical Runtime Config for Coding Agents
- Amazon Cognito Identity Pools: Federate Coding-Agent Tool Callers Without Long-Lived Keys
- Amazon Athena for Coding Agents: Query Tool Audit Logs and Eval Traces on S3 Without a Warehouse
- AWS Batch for Coding Agents: Parallel Overnight PR Reviews and Long-Running Jobs Without Lambda Timeouts
- Amazon Bedrock Prompt Management: Version and A/B Test Coding-Agent System Prompts
Deep-dive PDF
Get the expanded guide for this post — extra diagrams-style checklists, failure modes, and a production walkthrough. Free when you subscribe to CheatCoders.
Already subscribed? or open the subscribe page.
Discover more from CheatCoders
Subscribe to get the latest posts sent to your email.
