Node DNS Caching: Stop Micro-Outages From Brief Resolver Blips

Node DNS Caching: Stop Micro-Outages From Brief Resolver Blips

Node’s default DNS behavior is a silent reliability tax: dns.lookup hits the OS resolver per new connection family, and a 200ms CoreDNS blip becomes a thundering herd of ENOTFOUND / ETIMEOUT across every undici Agent. The unfair advantage is an application-level DNS cache with negative caching bounds, stale-while-revalidate, and explicit timeouts that fail individual lookups — not entire request batches.

⚡ TL;DR: Prefer CacheableLookup (or a thin TTL cache around dns.promises.lookup) wired into undici/http.Agent lookup; set positive TTL ≤ DNS TTL; negative TTL short (1–5s); never cache forever. Pair with Node undici Dispatcher Pools and HTTP/2 in Node Behind ALB.

Wire a cacheable lookup into undici

// src/dns-agent.ts
import CacheableLookup from "cacheable-lookup";
import { Agent, setGlobalDispatcher } from "undici";
import { Resolver } from "node:dns";

const resolver = new Resolver();
resolver.setServers(process.env.DNS_SERVERS?.split(",") ?? ["10.0.0.2"]);

const cacheable = new CacheableLookup({
  maxTtl: 30,          // ✅ clamp runaway TTLs
  errorTtl: 2,         // ✅ short negative cache
  resolver,
});

export const dispatcher = new Agent({
  connect: {
    lookup: cacheable.lookup.bind(cacheable) as any,
    timeout: 3_000,
  },
  keepAliveTimeout: 30_000,
  connections: 32,
});

setGlobalDispatcher(dispatcher);
// ❌ New Agent per request + default lookup — amplifies every blip
const res = await fetch(url); // default DNS, no shared cache

Stale-while-revalidate for critical peers

For in-mesh services with stable IPs, serve last-known-good briefly while refreshing.

// src/stale-dns.ts
type Entry = { address: string; family: 4 | 6; expires: number; staleUntil: number };
const map = new Map<string, Entry>();

export async function lookupCached(host: string, lookup: typeof import("node:dns").promises.lookup) {
  const now = Date.now();
  const hit = map.get(host);
  if (hit && hit.expires > now) return { address: hit.address, family: hit.family };
  try {
    const r = await lookup(host, { verbatim: true });
    map.set(host, {
      address: r.address,
      family: r.family as 4 | 6,
      expires: now + 15_000,
      staleUntil: now + 60_000,
    });
    return r;
  } catch (err) {
    // ✅ Serve stale for a bounded window instead of failing open forever
    if (hit && hit.staleUntil > now) {
      metrics.increment("dns.stale_served");
      return { address: hit.address, family: hit.family };
    }
    throw err;
  }
}

Timeouts and herd control

Knob Suggested start
Positive max TTL 15–30s (or DNS TTL)
Negative TTL 1–5s
Lookup timeout 1–3s
Stale window ≤60s for mesh peers only
Max concurrent lookups/host 1 in-flight + waiters
# ✅ Prove resolver latency in staging
dig @$COREDNS api.internal.svc.cluster.local +stats
# Alarm on dns.lookup error rate and p99 duration

Closing checklist

✅ Dos
– ✅ Share one cacheable lookup across the process dispatcher
– ✅ Bound negative caching so recoveries are fast
– ✅ Prefer stale-while-revalidate for critical mesh hosts
– ✅ Alarm DNS error rate separately from upstream 5xx
– ✅ Test behavior when CoreDNS is paused for 5s

❌ Don’ts
– ❌ Don’t cache DNS forever “for performance”
– ❌ Don’t ignore IPv6/verbatim surprises behind ALB
– ❌ Don’t create a new Agent (and lookup path) per request
– ❌ Don’t treat every ENOTFOUND as an application bug

Related reading

Last updated on September 11, 2026


Discover more from CheatCoders

Subscribe to get the latest posts sent to your email.

Comments

No comments yet. Why don’t you start the discussion?

Leave a Reply