20 views
This guide focuses on Node undici Dispatcher Pools for production systems, with practical trade-offs for reliability, security, and cost.
When traffic spikes, naive fetch usage opens a new connection per request until sockets pile up, DNS thrashes, and upstreams shed load. undici’s Agent / Pool is the control plane: cap connections, tune keep-alive, and fail fast on connect so one hot dependency cannot melt the process.
⚡ TL;DR: One sharedAgentper upstream origin; setconnections,pipelining,keepAliveTimeout, andconnect.timeout; disable keep-alive to broken peers; instrument pool stats. Pair with Node.js Event Loop Lag p99 and OpenTelemetry for LLMs.
Node undici Dispatcher Pools: production guidance
typescript
❌ // lib/http.ts
import { Agent, fetch as undiciFetch, setGlobalDispatcher } from "undici";
export const upstreamAgent = new Agent({
connections: 32, // ✅ hard cap per origin
pipelining: 1, // keep 1 unless origin proven safe
keepAliveTimeout: 10_000,
keepAliveMaxTimeout: 30_000,
connect: { timeout: 1_500, rejectUnauthorized: true },
bodyTimeout: 15_000,
headersTimeout: 10_000,
});
setGlobalDispatcher(upstreamAgent);
export function paymentsFetch(path: string, init?: RequestInit) {
return undiciFetch(`https://payments.internal${path}`, {
...init,
dispatcher: upstreamAgent,
});
}
new Agent() inside each handler (defeats pooling). ✅ Module-scope singleton per origin.
Stop storms at the edge
typescript
Combine with ALB slow-start and circuit breakers. When keep-alive connections are dead (LB idle timeout < client keep-alive), lower // middleware: shed before opening more sockets
let inFlight = 0;
const MAX = 200;
export async function guarded(path: string) {
if (inFlight >= MAX) {
const err = new Error("upstream_shed");
(err as any).status = 503;
throw err;
}
inFlight++;
try {
return await paymentsFetch(path);
} finally {
inFlight--;
}
}
keepAliveTimeout below the LB idle timeout — classic cause of “first request after idle fails.”
Observe the pool
typescript
Track connect timeouts vs response timeouts separately; connect storms look like rising connect failures with healthy upstream app metrics — see also Lambda Cold Starts when Node runs on Lambda behind the same ALB patterns.
// optional diagnostics
setInterval(() => {
// undici freeSockets / pending — expose via Prometheus gauge
console.log(JSON.stringify({ msg: "undici_stats", inFlight }));
}, 5_000).unref();
Closing checklist
✅ Dos – ✅ SingletonAgent/Pool per origin
– ✅ Cap connections; set connect + body timeouts
– ✅ Align keep-alive with LB idle timeouts
– ✅ Shed load when in-flight exceeds budget
– ✅ Separate metrics for connect vs response failures
❌ Don’ts
– ❌ Don’t create Agents per request
– ❌ Don’t leave default infinite-ish waits on connect
– ❌ Don’t enable high pipelining against HTTP/1.1 servers blindly
– ❌ Don’t ignore TLS session churn on short-lived clients
Related reading
- Node.js Event Loop Lag p99
- OpenTelemetry for LLMs
- Lambda Cold Starts on Node 20
- Lambda Powertools Node Structured Logs
Watch: 100 Days of AI on YouTube
Short videos from CheatCoders, one AI engineering topic per day.
Most viewed
- Python Decorators Explained: From Simple Wrappers to Production Patterns
- Claude Code vs Cursor vs GitHub Copilot: Choose by Workflow, and Set Up One Repo That Works With All Three
- AI Agent Frameworks in 2025: LangGraph vs CrewAI vs AutoGen vs Raw API
- Python String Methods: Every str Method With Real Production Examples
- Css Animation Performance: Performance-Optimized Motion for Modern UIs
Newly added
- AWS Systems Manager Parameter Store: Hierarchical Runtime Config for Coding Agents
- Amazon Cognito Identity Pools: Federate Coding-Agent Tool Callers Without Long-Lived Keys
- Amazon Athena for Coding Agents: Query Tool Audit Logs and Eval Traces on S3 Without a Warehouse
- AWS Batch for Coding Agents: Parallel Overnight PR Reviews and Long-Running Jobs Without Lambda Timeouts
- Amazon Bedrock Prompt Management: Version and A/B Test Coding-Agent System Prompts
Deep-dive PDF
Get the expanded guide for this post — extra diagrams-style checklists, failure modes, and a production walkthrough. Free when you subscribe to CheatCoders.
Already subscribed? or open the subscribe page.
Discover more from CheatCoders
Subscribe to get the latest posts sent to your email.
Pingback: uvloop With asyncio: When the Drop-In Speedup Stops Helping - CheatCoders