Lease-Based ECS Leadership: Singleton Cron That Survives Rolling Deploys

During a rolling ECS deploy you briefly have old and new tasks alive. If both believe they own the nightly billing cron, you double-charge. DynamoDB conditional leases give you fencing without ZooKeeper: one leader, TTL heartbeats, and a monotonic fencing token that makes stale leaders’ writes fail closed.

⚡ TL;DR: Elect with attribute_not_exists / TTL compare-and-swap on a lease row. Heartbeat under half TTL. Stamp every side effect with fencing token; reject lower tokens. Pair with Graceful Shutdown for Node and Idempotency Keys End-to-End.

Lease row and acquire

// lease/ddb-lease.ts
export async function tryAcquire(job: string, owner: string, ttlSec = 30) {
  const now = Math.floor(Date.now() / 1000);
  try {
    const out = await ddb.update({
      Key: { pk: `lease#${job}` },
      UpdateExpression:
        "SET #o = :o, #exp = :exp, fence = if_not_exists(fence, :z) + :one",
      ConditionExpression: "attribute_not_exists(pk) OR #exp < :now",
      ExpressionAttributeNames: { "#o": "owner", "#exp": "expiresAt" },
      ExpressionAttributeValues: {
        ":o": owner,
        ":exp": now + ttlSec,
        ":now": now,
        ":z": 0,
        ":one": 1,
      },
      ReturnValues: "ALL_NEW",
    });
    return { ok: true as const, fence: out.Attributes!.fence as number };
  } catch (e: any) {
    if (e.name === "ConditionalCheckFailedException") return { ok: false as const };
    throw e;
  }
}

Heartbeat and resign on SIGTERM

export async function heartbeat(job: string, owner: string, fence: number, ttlSec = 30) {
  const now = Math.floor(Date.now() / 1000);
  await ddb.update({
    Key: { pk: `lease#${job}` },
    UpdateExpression: "SET expiresAt = :exp",
    ConditionExpression: "owner = :o AND fence = :f",
    ExpressionAttributeValues: {
      ":exp": now + ttlSec,
      ":o": owner,
      ":f": fence,
    },
  }); // ✅ stale leader cannot extend
}

export function installResign(job: string, owner: string, fence: number) {
  process.on("SIGTERM", async () => {
    try {
      await ddb.update({
        Key: { pk: `lease#${job}` },
        UpdateExpression: "SET expiresAt = :z",
        ConditionExpression: "owner = :o AND fence = :f",
        ExpressionAttributeValues: { ":z": 0, ":o": owner, ":f": fence },
      });
    } finally {
      process.exit(0);
    }
  });
}

Fence every side effect

export async function runSingletonTick(job: string, fence: number, work: () => Promise<void>) {
  await ddb.put({
    Item: {
      pk: `jobrun#${job}#${slotId()}`,
      fence,
      status: "started",
    },
    ConditionExpression: "attribute_not_exists(pk)",
  });
  await work();
}

export async function applySideEffect(evt: { fence: number; payload: unknown }) {
  const lease = await ddb.get({ Key: { pk: `lease#billing-nightly` } });
  if (!lease.Item || evt.fence < lease.Item.fence) {
    // ❌ stale leader after deploy — fail closed
    throw new Error("stale_fence");
  }
  await sink.write(evt.payload);
}

Deploy timing rules

Prefer EventBridge Scheduler invoking a shared API that acquires the lease, rather than every ECS task waking on node-cron.

Closing checklist

✅ Dos
– ✅ Conditional acquire with TTL expiry
– ✅ Heartbeat under ConditionExpression on owner+fence
– ✅ Resign on SIGTERM by zeroing expiry
– ✅ Stamp side effects with fencing token
– ✅ Alarm when lease flips more than N times/hour

❌ Don’ts
– ❌ Don’t run node-cron on every task without a lease
– ❌ Don’t trust wall-clock alone without fencing tokens
– ❌ Don’t set TTL shorter than deploy drain time without resign
– ❌ Don’t ignore ConditionalCheckFailed on heartbeat
– ❌ Don’t dual-write “who is leader” to local memory only

Related reading

• Graceful Shutdown for Node: Drain Correctly on Kubernetes and ECS
• Idempotency Keys End-to-End: API Gateway Through Step Functions Safely
• Exactly-Once Illusions: Design At-Least-Once Plus Truly Idempotent Handlers
• Saga Compensations Under Partial Failure

Last updated on September 11, 2026