Node undici Dispatcher Pools: Kill Keep-Alive Storms Under Spikes

When traffic spikes, naive fetch usage opens a new connection per request until sockets pile up, DNS thrashes, and upstreams shed load. undici’s Agent / Pool is the control plane: cap connections, tune keep-alive, and fail fast on connect so one hot dependency cannot melt the process.

⚡ TL;DR: One shared Agent per upstream origin; set connections, pipelining, keepAliveTimeout, and connect.timeout; disable keep-alive to broken peers; instrument pool stats. Pair with Node.js Event Loop Lag p99 and OpenTelemetry for LLMs.

Shared Agent, not ad-hoc fetch

// lib/http.ts
import { Agent, fetch as undiciFetch, setGlobalDispatcher } from "undici";

export const upstreamAgent = new Agent({
  connections: 32,           // ✅ hard cap per origin
  pipelining: 1,             // keep 1 unless origin proven safe
  keepAliveTimeout: 10_000,
  keepAliveMaxTimeout: 30_000,
  connect: { timeout: 1_500, rejectUnauthorized: true },
  bodyTimeout: 15_000,
  headersTimeout: 10_000,
});

setGlobalDispatcher(upstreamAgent);

export function paymentsFetch(path: string, init?: RequestInit) {
  return undiciFetch(`https://payments.internal${path}`, {
    ...init,
    dispatcher: upstreamAgent,
  });
}

❌ new Agent() inside each handler (defeats pooling). ✅ Module-scope singleton per origin.

Stop storms at the edge

// middleware: shed before opening more sockets
let inFlight = 0;
const MAX = 200;

export async function guarded(path: string) {
  if (inFlight >= MAX) {
    const err = new Error("upstream_shed");
    (err as any).status = 503;
    throw err;
  }
  inFlight++;
  try {
    return await paymentsFetch(path);
  } finally {
    inFlight--;
  }
}

Combine with ALB slow-start and circuit breakers. When keep-alive connections are dead (LB idle timeout < client keep-alive), lower keepAliveTimeout below the LB idle timeout — classic cause of “first request after idle fails.”

Observe the pool

// optional diagnostics
setInterval(() => {
  // undici freeSockets / pending — expose via Prometheus gauge
  console.log(JSON.stringify({ msg: "undici_stats", inFlight }));
}, 5_000).unref();

Track connect timeouts vs response timeouts separately; connect storms look like rising connect failures with healthy upstream app metrics — see also Lambda Cold Starts when Node runs on Lambda behind the same ALB patterns.

Closing checklist

✅ Dos
– ✅ Singleton Agent/Pool per origin
– ✅ Cap connections; set connect + body timeouts
– ✅ Align keep-alive with LB idle timeouts
– ✅ Shed load when in-flight exceeds budget
– ✅ Separate metrics for connect vs response failures

❌ Don’ts
– ❌ Don’t create Agents per request
– ❌ Don’t leave default infinite-ish waits on connect
– ❌ Don’t enable high pipelining against HTTP/1.1 servers blindly
– ❌ Don’t ignore TLS session churn on short-lived clients

Related reading

• Node.js Event Loop Lag p99
• OpenTelemetry for LLMs
• Lambda Cold Starts on Node 20
• Lambda Powertools Node Structured Logs

Last updated on September 11, 2026