Zero-Copy Node Streams: Pipe Large S3 Objects Without Buffering

await Body.transformToByteArray() on a 8GB object is how Node services OOM in production. The correct pattern is pipeline() from S3’s readable stream through transforms into the destination (HTTP response, another PutObject, or disk) while honoring backpressure and verifying checksums incrementally. “Zero-copy” here means zero extra full-object buffers — not magic without copies in V8.

⚡ TL;DR: Use @aws-sdk/client-s3 with Body as Readable; wrap in pipeline with createHash transforms; set highWaterMark deliberately; never concatenate chunks into one Buffer for large objects. Cross-link: Brotli vs Gzip in Node for streaming compression and Lambda + Bedrock streaming for the same backpressure mindset.

The anti-pattern and the pipeline

// ❌ Full object in memory
import { S3Client, GetObjectCommand } from "@aws-sdk/client-s3";
const s3 = new S3Client({});
const out = await s3.send(new GetObjectCommand({ Bucket, Key }));
const bytes = await out.Body!.transformToByteArray(); // boom on large keys

// ✅ Stream with backpressure + incremental checksum
import { S3Client, GetObjectCommand, PutObjectCommand } from "@aws-sdk/client-s3";
import { Upload } from "@aws-sdk/lib-storage";
import { pipeline } from "node:stream/promises";
import { createHash } from "node:crypto";
import { createGunzip } from "node:zlib";
import { Readable, Transform } from "node:stream";

const s3 = new S3Client({});

function hashTap(algo: string) {
  const h = createHash(algo);
  return new Transform({
    transform(chunk, _e, cb) {
      h.update(chunk);
      cb(null, chunk);
    },
    flush(cb) {
      (this as any).digestHex = h.digest("hex");
      cb();
    },
  });
}

export async function mirrorS3(srcBucket: string, key: string, dstBucket: string) {
  const got = await s3.send(new GetObjectCommand({ Bucket: srcBucket, Key: key }));
  const body = got.Body as Readable;
  const tap = hashTap("sha256");

  const upload = new Upload({
    client: s3,
    params: {
      Bucket: dstBucket,
      Key: key,
      Body: body.pipe(tap), // still prefer pipeline() below for error propagation
      ContentType: got.ContentType,
    },
    queueSize: 16 * 1024 * 1024,
  });

  // Better: explicit pipeline into a PassThrough feeding Upload
  await upload.done();
  const digest = (tap as any).digestHex as string;
  if (got.ChecksumSHA256 && got.ChecksumSHA256 !== digest) {
    // Note: S3 checksum headers may be base64 — normalize before compare
    throw new Error("checksum_mismatch");
  }
  return digest;
}

pipeline() for HTTP egress

import { pipeline } from "node:stream/promises";
import type { IncomingMessage, ServerResponse } from "node:http";

export async function downloadHandler(req: IncomingMessage, res: ServerResponse, key: string) {
  const got = await s3.send(new GetObjectCommand({ Bucket: process.env.BUCKET!, Key: key }));
  res.writeHead(200, {
    "Content-Type": got.ContentType ?? "application/octet-stream",
    "Content-Length": got.ContentLength ? String(got.ContentLength) : undefined,
    ETag: got.ETag,
  });
  await pipeline(got.Body as Readable, res);
}

highWaterMark and concurrency

Default chunking can still blow RSS if you fan out dozens of parallel pipelines. Cap concurrent large transfers per process and tune highWaterMark (often 64KB–1MB) based on profile data from Node Flamegraphs on ECS.

import PQueue from "p-queue";
const queue = new PQueue({ concurrency: 4 });

export function enqueueMirror(job: () => Promise<void>) {
  return queue.add(job);
}

Checksums and multipart

• Prefer S3 object checksums (CRC32C/SHA256) and validate when present.
• For multipart uploads, use @aws-sdk/lib-storage Upload so parts stream without a full local file.
• On gunzip transforms, verify uncompressed size against an allowlist to avoid zip-bombs.

export async function readGunzipLimited(src: Readable, maxBytes: number) {
  let n = 0;
  const limiter = new Transform({
    transform(chunk, _e, cb) {
      n += chunk.length;
      if (n > maxBytes) cb(new Error("uncompressed_limit"));
      else cb(null, chunk);
    },
  });
  // pipeline(src, createGunzip(), limiter, dest)
  return { limiter };
}

Closing checklist

✅ Dos
– ✅ pipeline() from GetObject Body to destination
– ✅ Incremental hashes / S3 checksum verification
– ✅ Cap concurrent large transfers
– ✅ Use multipart Upload for big PutObject paths
– ✅ Bound decompression sizes

❌ Don’ts
– ❌ Don’t transformToByteArray() / Buffer.concat multi-GB bodies
– ❌ Don’t ignore pipeline errors (orphaned sockets)
– ❌ Don’t fan out unlimited parallel mirrors
– ❌ Don’t trust client-provided Content-Length alone
– ❌ Don’t disable backpressure by buffering in custom transforms

Related reading

• Brotli vs Gzip in Node: CPU Tradeoffs at the CDN Edge
• Lambda + Bedrock: Stream Tokens Without API Gateway Caps
• Node Flamegraphs on ECS: Continuous Profiling That Survives Scale
• Node Fetch vs Got vs Undici: Throughput Under Real TLS Churn

Last updated on September 11, 2026