Blue/Green Stateful Workers: Drain In-Flight Jobs Before Old Tasks Die

Stateless HTTP blue/green is easy: drain connections. Stateful queue workers die messily—SIGTERM mid-batch duplicates SQS messages or abandons Kinesis checkpoints. The unfair advantage is a drain protocol: stop pulling, finish in-flight with a deadline, checkpoint, then exit—wired into ECS deployment circuit breakers.

⚡ TL;DR: On SIGTERM set accepting=false; stop ReceiveMessage / shard lease renewals; await in-flight with timeout < stopTimeout; delete/checkpoint only after success; size ECS stopTimeout above max job duration. Pair with Graceful Shutdown for Node and Lambda SQS Partial Batch Failures.

The drain state machine

RUNNING → DRAINING (SIGTERM) → IDLE (in-flight=0) → EXIT 0
                ↘ DEADLINE → nack/extend visibility → EXIT 1 (deploy retries)

// ✅ Node SQS worker drain
import { Consumer } from "sqs-consumer";

let accepting = true;
const inFlight = new Set<Promise<void>>();

const app = Consumer.create({
  queueUrl: process.env.QUEUE_URL!,
  handleMessage: async (msg) => {
    if (!accepting) throw new Error("draining"); // ✅ don't start new work
    const p = processMessage(msg).finally(() => inFlight.delete(p));
    inFlight.add(p);
    await p;
  },
  bodyParser: JSON.parse,
});

async function shutdown(signal: string) {
  console.info({ signal, msg: "drain_start", inFlight: inFlight.size });
  accepting = false;
  app.stop(); // stop polling
  const deadline = Date.now() + 25_000; // < ECS stopTimeout (30s)
  while (inFlight.size && Date.now() < deadline) {
    await Promise.race([...inFlight, sleep(200)]);
  }
  if (inFlight.size) {
    console.error({ left: inFlight.size, msg: "drain_timeout" });
    process.exit(1); // ✅ visibility timeout will redeliver
  }
  process.exit(0);
}

process.on("SIGTERM", () => void shutdown("SIGTERM"));
app.start();

Kinesis / EventBridge considerations

# ✅ ECS task def: stopTimeout seconds must exceed drain deadline
# stopTimeout: 60
# ❌ default 30s with 45s video transcodes → SIGKILL every deploy

Blue/green cutover order

1. Start green tasks; wait healthy.
2. Shift queue consumers by draining blue (desired count ↓) while green polls.
3. For singleton lease workers, transfer DynamoDB lease before killing blue (Lease-Based ECS Leadership).
4. Verify DLQ rate and lag gauges flat across the cut.

❌ Cutting load balancer first on a dual HTTP+SQS task — HTTP drains while SQS still receives until SIGTERM.

Closing checklist

✅ Dos
– ✅ Gate new work on an accepting flag
– ✅ Track in-flight promises/goroutines explicitly
– ✅ Set stopTimeout > drain deadline > p99 job time
– ✅ Make handlers idempotent for inevitable duplicates
– ✅ Alert on drain timeouts per deploy

❌ Don’ts
– ❌ Don’t ack before side effects commit
– ❌ Don’t ignore SIGTERM (ECS will SIGKILL)
– ❌ Don’t share one process for HTTP and multi-minute jobs without separate drain budgets
– ❌ Don’t shrink visibility timeout below max processing time

Related reading

• Graceful Shutdown for Node: Drain Correctly on Kubernetes and ECS
• Lambda SQS Partial Batch Failures
• Lease-Based ECS Leadership
• Kafka to Lambda Backpressure

Last updated on September 11, 2026