10 views
During a rolling ECS deploy you briefly have old and new tasks alive. If both believe they own the nightly billing cron, you double-charge. DynamoDB conditional leases give you fencing without ZooKeeper: one leader, TTL heartbeats, and a monotonic fencing token that makes stale leaders’ writes fail closed.
⚡ TL;DR: Elect with attribute_not_exists / TTL compare-and-swap on a lease row. Heartbeat under half TTL. Stamp every side effect with fencing token; reject lower tokens. Pair with Graceful Shutdown for Node and Idempotency Keys End-to-End.
Lease-Based ECS Leadership: production guidance
typescript
// lease/ddb-lease.ts
export async function tryAcquire(job: string, owner: string, ttlSec = 30) {
const now = Math.floor(Date.now() / 1000);
try {
const out = await ddb.update({
Key: { pk: `lease#${job}` },
UpdateExpression:
"SET #o = :o, #exp = :exp, fence = if_not_exists(fence, :z) + :one",
ConditionExpression: "attribute_not_exists(pk) OR #exp < :now",
ExpressionAttributeNames: { "#o": "owner", "#exp": "expiresAt" },
ExpressionAttributeValues: {
":o": owner,
":exp": now + ttlSec,
":now": now,
":z": 0,
":one": 1,
},
ReturnValues: "ALL_NEW",
});
return { ok: true as const, fence: out.Attributes!.fence as number };
} catch (e: any) {
if (e.name === "ConditionalCheckFailedException") return { ok: false as const };
throw e;
}
}
Heartbeat and resign on SIGTERM
typescript
export async function heartbeat(job: string, owner: string, fence: number, ttlSec = 30) {
const now = Math.floor(Date.now() / 1000);
await ddb.update({
Key: { pk: `lease#${job}` },
UpdateExpression: "SET expiresAt = :exp",
ConditionExpression: "owner = :o AND fence = :f",
ExpressionAttributeValues: {
":exp": now + ttlSec,
":o": owner,
":f": fence,
},
}); // ✅ stale leader cannot extend
}
export function installResign(job: string, owner: string, fence: number) {
process.on("SIGTERM", async () => {
try {
await ddb.update({
Key: { pk: `lease#${job}` },
UpdateExpression: "SET expiresAt = :z",
ConditionExpression: "owner = :o AND fence = :f",
ExpressionAttributeValues: { ":z": 0, ":o": owner, ":f": fence },
});
} finally {
process.exit(0);
}
});
}
Fence every side effect
typescript
export async function runSingletonTick(job: string, fence: number, work: () => Promise<void>) {
await ddb.put({
Item: {
pk: `jobrun#${job}#${slotId()}`,
fence,
status: "started",
},
ConditionExpression: "attribute_not_exists(pk)",
});
await work();
}
export async function applySideEffect(evt: { fence: number; payload: unknown }) {
const lease = await ddb.get({ Key: { pk: `lease#billing-nightly` } });
if (!lease.Item || evt.fence < lease.Item.fence) {
// ❌ stale leader after deploy — fail closed
throw new Error("stale_fence");
}
await sink.write(evt.payload);
}
Deploy timing rules
| Knob | Guidance |
|---|---|
| Lease TTL | ≥ 2× heartbeat interval; ≤ 60s typical |
stopTimeout |
> heartbeat interval so resign can run |
| minHealthyPercent | Prefer 100/200 for singleton services |
| Cron source | EventBridge → leader check, not every task fires work |
node-cron.
Closing checklist
✅ Dos – ✅ Conditional acquire with TTL expiry – ✅ Heartbeat under ConditionExpression on owner+fence – ✅ Resign on SIGTERM by zeroing expiry – ✅ Stamp side effects with fencing token – ✅ Alarm when lease flips more than N times/hour ❌ Don’ts – ❌ Don’t runnode-cron on every task without a lease
– ❌ Don’t trust wall-clock alone without fencing tokens
– ❌ Don’t set TTL shorter than deploy drain time without resign
– ❌ Don’t ignore ConditionalCheckFailed on heartbeat
– ❌ Don’t dual-write “who is leader” to local memory only
Related reading
- Graceful Shutdown for Node: Drain Correctly on Kubernetes and ECS
- Idempotency Keys End-to-End: API Gateway Through Step Functions Safely
- Exactly-Once Illusions: Design At-Least-Once Plus Truly Idempotent Handlers
- Saga Compensations Under Partial Failure
Watch: 100 Days of AI on YouTube
Short videos from CheatCoders, one AI engineering topic per day.
Most viewed
- Python Decorators Explained: From Simple Wrappers to Production Patterns
- Claude Code vs Cursor vs GitHub Copilot: Choose by Workflow, and Set Up One Repo That Works With All Three
- AI Agent Frameworks in 2025: LangGraph vs CrewAI vs AutoGen vs Raw API
- Python String Methods: Every str Method With Real Production Examples
- Css Animation Performance: Performance-Optimized Motion for Modern UIs
Newly added
- AWS Systems Manager Parameter Store: Hierarchical Runtime Config for Coding Agents
- Amazon Cognito Identity Pools: Federate Coding-Agent Tool Callers Without Long-Lived Keys
- Amazon Athena for Coding Agents: Query Tool Audit Logs and Eval Traces on S3 Without a Warehouse
- AWS Batch for Coding Agents: Parallel Overnight PR Reviews and Long-Running Jobs Without Lambda Timeouts
- Amazon Bedrock Prompt Management: Version and A/B Test Coding-Agent System Prompts
Deep-dive PDF
Get the expanded guide for this post — extra diagrams-style checklists, failure modes, and a production walkthrough. Free when you subscribe to CheatCoders.
Already subscribed? or open the subscribe page.
Discover more from CheatCoders
Subscribe to get the latest posts sent to your email.

Pingback: Distributed Locks Reality Check: When Redis Redlock Is the Wrong Tool - CheatCoders