Python Prefork Pitfalls: Inherited Sockets, RNGs, and Pool Re-Init

Gunicorn/multiprocessing prefork copies the parent address space. That includes open DB sockets, SSL sessions, boto3 clients, logging listeners, and PRNG state. Two workers then fight over one TCP connection or emit identical “random” tokens. The fix is ritualized: close or reset everything that is not fork-safe in post_fork (or never create it before fork).

⚡ TL;DR: Create connection pools after fork; re-seed random/secrets/NumPy RNGs in each worker; close inherited sockets; restart QueueListener threads. SnapStart and Lambda freeze/restore have the same class of bugs—see Lambda SnapStart for Python: Pitfalls Beyond the Java Marketing. Worker model context: Python asyncio vs Threading vs Multiprocessing.

What fork duplicates (badly)

# ❌ Created in master before fork
import boto3
from sqlalchemy import create_engine

engine = create_engine(os.environ["DATABASE_URL"], pool_size=5)
s3 = boto3.client("s3")
# workers inherit same pool connections → protocol chaos

post_fork: dispose and rebuild

# gunicorn_conf.py
import os, random, logging

def post_fork(server, worker):
    # RNGs
    random.seed(int.from_bytes(os.urandom(16), "big"))
    try:
        import numpy as np
        np.random.seed(None)
    except ImportError:
        pass

    # SQLAlchemy
    from app.db import engine
    engine.dispose(close=True)  # ✅ drop inherited conns; pool recreates lazily

    # HTTP / AWS clients: rebuild
    from app import clients
    clients.reset()

    # Logging listeners (threads do not survive fork cleanly)
    from app.logging_setup import configure_async_logging
    configure_async_logging()

    logging.getLogger(__name__).info("worker_booted pid=%s", worker.pid)

# app/clients.py
import boto3

_s3 = None

def reset():
    global _s3
    _s3 = boto3.client("s3")

def s3():
    if _s3 is None:
        reset()
    return _s3

Inherited sockets and listen FDs

The master should own the listen socket; workers accept after fork—that part is fine. Client connections opened during master preload are not. Also watch:

• Redis/hiredis connections
• gRPC channels
• OpenTelemetry exporters / background BatchSpanProcessor threads
• multiprocessing.Queue feeder threads

# ✅ Lazy init pattern
class Pool:
    def __init__(self):
        self._redis = None

    def redis(self):
        if self._redis is None:
            import redis
            self._redis = redis.Redis.from_url(os.environ["REDIS_URL"])
        return self._redis

ECS/K8s vs Gunicorn

On ECS Fargate you often run one process per task (no prefork)—simpler. If you still prefork inside the task for CPU-bound workers, apply the same post_fork checklist. Graceful shutdown must drain the worker before the master exits—mirror Node drain patterns in Graceful Shutdown for Node when available, or your orchestrator’s preStop hook.

Closing checklist

✅ Dos
– ✅ engine.dispose() / close pools in post_fork
– ✅ Re-seed all RNGs per worker
– ✅ Rebuild SDK clients and OTEL processors after fork
– ✅ Prefer lazy init for sockets created on first use
– ✅ Log worker pid on boot for correlation

❌ Don’ts
– ❌ Don’t open DB/Redis connections in the master preload without dispose
– ❌ Don’t assume threads survive fork
– ❌ Don’t share SSL sessions across workers
– ❌ Don’t generate idempotency keys with a forked PRNG
– ❌ Don’t ignore silent duplicate side effects after deploy

Related reading

• Lambda SnapStart for Python: Pitfalls Beyond the Java Marketing
• Python asyncio vs Threading vs Multiprocessing
• Lambda Warm Pools: Low-Latency Backends for Coding Agent Tools
• Human-in-the-Loop Gates: Dual Control for Prod-Touching Agent Tools

Last updated on September 11, 2026