Lease-Based ECS Leadership: Singleton Cron That Survives Rolling Deploys

Lease-Based ECS Leadership: Singleton Cron That Survives Rolling Deploys
10 views
During a rolling ECS deploy you briefly have old and new tasks alive. If both believe they own the nightly billing cron, you double-charge. DynamoDB conditional leases give you fencing without ZooKeeper: one leader, TTL heartbeats, and a monotonic fencing token that makes stale leaders’ writes fail closed.
⚡ TL;DR: Elect with attribute_not_exists / TTL compare-and-swap on a lease row. Heartbeat under half TTL. Stamp every side effect with fencing token; reject lower tokens. Pair with Graceful Shutdown for Node and Idempotency Keys End-to-End.

Lease-Based ECS Leadership: production guidance

typescript
// lease/ddb-lease.ts
export async function tryAcquire(job: string, owner: string, ttlSec = 30) {
 const now = Math.floor(Date.now() / 1000);
 try {
 const out = await ddb.update({
 Key: { pk: `lease#${job}` },
 UpdateExpression:
 "SET #o = :o, #exp = :exp, fence = if_not_exists(fence, :z) + :one",
 ConditionExpression: "attribute_not_exists(pk) OR #exp < :now",
 ExpressionAttributeNames: { "#o": "owner", "#exp": "expiresAt" },
 ExpressionAttributeValues: {
 ":o": owner,
 ":exp": now + ttlSec,
 ":now": now,
 ":z": 0,
 ":one": 1,
 },
 ReturnValues: "ALL_NEW",
 });
 return { ok: true as const, fence: out.Attributes!.fence as number };
 } catch (e: any) {
 if (e.name === "ConditionalCheckFailedException") return { ok: false as const };
 throw e;
 }
}

Heartbeat and resign on SIGTERM

typescript
export async function heartbeat(job: string, owner: string, fence: number, ttlSec = 30) {
 const now = Math.floor(Date.now() / 1000);
 await ddb.update({
 Key: { pk: `lease#${job}` },
 UpdateExpression: "SET expiresAt = :exp",
 ConditionExpression: "owner = :o AND fence = :f",
 ExpressionAttributeValues: {
 ":exp": now + ttlSec,
 ":o": owner,
 ":f": fence,
 },
 }); // ✅ stale leader cannot extend
}

export function installResign(job: string, owner: string, fence: number) {
 process.on("SIGTERM", async () => {
 try {
 await ddb.update({
 Key: { pk: `lease#${job}` },
 UpdateExpression: "SET expiresAt = :z",
 ConditionExpression: "owner = :o AND fence = :f",
 ExpressionAttributeValues: { ":z": 0, ":o": owner, ":f": fence },
 });
 } finally {
 process.exit(0);
 }
 });
}

Fence every side effect

typescript
export async function runSingletonTick(job: string, fence: number, work: () => Promise<void>) {
 await ddb.put({
 Item: {
 pk: `jobrun#${job}#${slotId()}`,
 fence,
 status: "started",
 },
 ConditionExpression: "attribute_not_exists(pk)",
 });
 await work();
}

export async function applySideEffect(evt: { fence: number; payload: unknown }) {
 const lease = await ddb.get({ Key: { pk: `lease#billing-nightly` } });
 if (!lease.Item || evt.fence < lease.Item.fence) {
 // ❌ stale leader after deploy — fail closed
 throw new Error("stale_fence");
 }
 await sink.write(evt.payload);
}

Deploy timing rules

Knob Guidance
Lease TTL ≥ 2× heartbeat interval; ≤ 60s typical
stopTimeout > heartbeat interval so resign can run
minHealthyPercent Prefer 100/200 for singleton services
Cron source EventBridge → leader check, not every task fires work
Prefer EventBridge Scheduler invoking a shared API that acquires the lease, rather than every ECS task waking on node-cron.

Closing checklist

✅ Dos – ✅ Conditional acquire with TTL expiry – ✅ Heartbeat under ConditionExpression on owner+fence – ✅ Resign on SIGTERM by zeroing expiry – ✅ Stamp side effects with fencing token – ✅ Alarm when lease flips more than N times/hour ❌ Don’ts – ❌ Don’t run node-cron on every task without a lease – ❌ Don’t trust wall-clock alone without fencing tokens – ❌ Don’t set TTL shorter than deploy drain time without resign – ❌ Don’t ignore ConditionalCheckFailed on heartbeat – ❌ Don’t dual-write “who is leader” to local memory only

Related reading

Last updated on September 11, 2026 Related: Node Flamegraphs on ECS: Continuous Profiling That Survives Scale; AWS Multi-Tenant SaaS Control Plane Architecture: API Gateway, ECS, DynamoDB, Step Functions, and EventBridge; Git Branching Strategies: GitFlow, Trunk-Based, and GitHub Flow Compared.

Watch: 100 Days of AI on YouTube

Short videos from CheatCoders, one AI engineering topic per day.

Open the playlist on YouTube · CheatCoders on YouTube

Deep-dive PDF

Get the expanded guide for this post — extra diagrams-style checklists, failure modes, and a production walkthrough. Free when you subscribe to CheatCoders.

Already subscribed? or open the subscribe page.


Discover more from CheatCoders

Subscribe to get the latest posts sent to your email.

1 Comment

Leave a comment

No account needed. Name and email are optional.