Posted inAI AWS System Design

EventBridge Scheduler: Overnight Agent Batches Without Cron Drift

No Comments
12 views
You scheduled a “nightly” RAG reindex with an EventBridge rule cron: cron(30 23 * * ? *) in UTC. Someone assumed that meant 11:30 PM in India. Six months later the job runs at the wrong wall-clock time, overlaps the morning traffic spike, and a missed invoke from a transient Lambda throttle gets replayed into a double reindex that corrupts the vector store. The unfair advantage for overnight agent batches is EventBridge Scheduler — first-class timezones, flexible windows, one-time and rate schedules, and dead-lettering — plus idempotency tokens so a late delivery never double-applies.
⚡ TL;DR: Prefer EventBridge Scheduler over classic EventBridge rule cron for overnight agent work (RAG reindex, eval harnesses, batch Converse). Set ScheduleExpressionTimezone to Asia/Kolkata, use flexible time windows, wire a DLQ, and stamp every run with an idempotency key derived from the schedule window. Pair with Bedrock Agents + EventBridge Async Tools, EventBridge Pipes to Lambda, and SQS FIFO Ordered Agent Job Queues.

EventBridge Scheduler: production guidance

EventBridge rules evaluate cron in UTC unless you carefully document otherwise. Humans think in local time. Overnight agent batches usually need:
  • Wall-clock alignment with team SLAs (“after 1 AM IST, before 5 AM IST”)
  • Catch-up without stampeding when the account was throttled
  • Exactly-once business effect even if the target is invoked twice
Rules also lack first-class flexible time windows and one-time schedules with automatic completion. Scheduler was built for that.
typescript
// ❌ Classic rule: UTC cron, easy to misread as local
{
 ScheduleExpression: "cron(30 19 * * ? *)", // "7:30 PM UTC" — is that IST?
 // No timezone field, no flexible window, no schedule-level DLQ of the same shape
}
✅ Scheduler: cron(30 1 * * ? *) with ScheduleExpressionTimezone: Asia/Kolkata.

One-time vs rate vs cron schedules

Kind When to use for agents Example
at() one-time Backfill a single eval run, “run this reindex at 02:00 tomorrow” at(2026-09-19T02:00:00)
rate() Steady heartbeat (every 15 minutes health / cheap sync) rate(15 minutes)
cron() Nightly / weekly agent batches cron(0 2 * * ? *) + timezone
For overnight RAG and eval, prefer cron + timezone. Use one-time schedules for operator-triggered backfills so you do not leave a permanent rule that someone forgets.
typescript
import {
 SchedulerClient,
 CreateScheduleCommand,
} from "@aws-sdk/client-scheduler";

const scheduler = new SchedulerClient({});

export async function createNightlyRagReindex(input: {
 scheduleName: string;
 lambdaArn: string;
 roleArn: string;
 dlqArn: string;
}) {
 await scheduler.send(
 new CreateScheduleCommand({
 Name: input.scheduleName,
 ScheduleExpression: "cron(0 2 * * ? *)", // 02:00
 ScheduleExpressionTimezone: "Asia/Kolkata", // ✅ wall clock IST
 FlexibleTimeWindow: {
 Mode: "FLEXIBLE",
 MaximumWindowInMinutes: 30, // ✅ avoid thundering herd at :00
 },
 Target: {
 Arn: input.lambdaArn,
 RoleArn: input.roleArn,
 Input: JSON.stringify({
 jobType: "rag_reindex",
 // static template; runtime adds window id below in handler
 }),
 DeadLetterConfig: { Arn: input.dlqArn },
 RetryPolicy: {
 MaximumEventAgeInSeconds: 3600,
 MaximumRetryAttempts: 2,
 },
 },
 State: "ENABLED",
 })
 );
}

Flexible time windows and Asia/Kolkata

A flexible window of 15–30 minutes lets Scheduler spread invocations so you do not synchronize every account’s nightly job to the same second. For agent batches that hit Bedrock or shared OpenSearch, that alone prevents self-inflicted throttle storms. Timezone Asia/Kolkata has no DST, which removes one class of “cron drift” — but document the timezone on the schedule name and in runbooks. Future you will assume UTC.
typescript
// ✅ Name encodes timezone intent
const name = "agent-rag-reindex-0200-ist";

Idempotency so missed runs do not double-apply

Scheduler retries and DLQ redrives can deliver the same logical night twice. Your Lambda must treat “Tuesday IST reindex” as a single business operation. Derive a stable key from the schedule name + local date of the intended window, not from Date.now().
typescript
import { createHash } from "crypto";
import { DynamoDBClient } from "@aws-sdk/client-dynamodb";
import {
 DynamoDBDocumentClient,
 PutCommand,
} from "@aws-sdk/lib-dynamodb";

const ddb = DynamoDBDocumentClient.from(new DynamoDBClient({}));

function istDateString(d = new Date()): string {
 // en-CA gives YYYY-MM-DD; timeZone forces Asia/Kolkata calendar date
 return new Intl.DateTimeFormat("en-CA", {
 timeZone: "Asia/Kolkata",
 year: "numeric",
 month: "2-digit",
 day: "2-digit",
 }).format(d);
}

export async function claimNightlyRun(scheduleName: string): Promise<{
 claimed: boolean;
 idempotencyKey: string;
}> {
 const day = istDateString();
 const idempotencyKey = createHash("sha256")
 .update(`${scheduleName}#${day}`)
 .digest("hex")
 .slice(0, 32);

 try {
 await ddb.send(
 new PutCommand({
 TableName: process.env.IDEMPOTENCY_TABLE!,
 Item: {
 pk: `job#${idempotencyKey}`,
 sk: "claim",
 scheduleName,
 day,
 status: "running",
 ttl: Math.floor(Date.now() / 1000) + 86400 * 7,
 },
 ConditionExpression: "attribute_not_exists(pk)",
 })
 );
 return { claimed: true, idempotencyKey };
 } catch (e: any) {
 if (e.name === "ConditionalCheckFailedException") {
 return { claimed: false, idempotencyKey };
 }
 throw e;
 }
}

export async function handler(event: { jobType: string }) {
 const scheduleName = process.env.SCHEDULE_NAME ?? "agent-rag-reindex-0200-ist";
 const { claimed, idempotencyKey } = await claimNightlyRun(scheduleName);
 if (!claimed) {
 // ✅ Late retry / double delivery — no-op
 return { ok: true, skipped: true, idempotencyKey };
 }

 // run RAG reindex / eval / batch Converse with idempotencyKey in logs + EMF
 await runAgentBatch({ jobType: event.jobType, idempotencyKey });
 return { ok: true, skipped: false, idempotencyKey };
}
Same pattern pairs with Bedrock Converse toolConfig idempotent tool results when the overnight job itself drives tool-using agents.

Scheduler vs EventBridge rules — when to keep rules

Keep rules when you need event-pattern matching (S3 Object Created → agent pipeline) or bus fan-out. Use Scheduler when time is the primary trigger.
Concern Rules cron Scheduler
Timezone UTC mindset Explicit ScheduleExpressionTimezone
Flexible window No Yes
One-time fire-and-forget Awkward Native at()
Schedule-level DLQ / retry Via target mostly First-class on target
Overnight agent batches Easy to get wrong Default choice
For async agent tools that should not block a Bedrock session, combine Scheduler (time) with EventBridge buses (facts) — do not force everything through one cron.

Dead-letter and observability

Always set DeadLetterConfig on the schedule target. Alarm on DLQ depth and on “skipped due to idempotency” vs “failed before claim” so you can tell catch-up from poison messages. Emit EMF with dimensions jobType, scheduleName, day (IST). If a night is skipped because the claim already exists, that is a success metric — not an error — unless the first run never finished (store status=running with a watchdog).
typescript
// ✅ Watchdog: if status=running older than 2h, allow reclaim or page

Checklist

  • [ ] Overnight jobs use EventBridge Scheduler, not UTC-only rule cron
  • [ ] ScheduleExpressionTimezone = Asia/Kolkata (or your real ops TZ)
  • [ ] Flexible window 15–30 minutes for Bedrock / shared index workloads
  • [ ] Target DLQ + low retry count; alarm on DLQ visible messages
  • [ ] Idempotency key = scheduleName + IST calendar date claimed in DynamoDB
  • [ ] Double delivery returns skipped success, not a second reindex
  • [ ] One-time at() for backfills; cron for recurring nights
  • [ ] Runbooks state timezone explicitly next to the cron expression
Cron drift is a people problem disguised as infrastructure. Scheduler makes the timezone explicit; idempotency makes retries safe. Overnight agent batches need both. Related: Day 34: EventBridge as the Agent’s Async Backbone; Multi-Agent Systems: How to Build Agent Teams That Do Not Hallucinate Each Other Into Failure; AI Agent Frameworks in 2025: LangGraph vs CrewAI vs AutoGen vs Raw API.

Watch: 100 Days of AI on YouTube

Short videos from CheatCoders, one AI engineering topic per day.

Open the playlist on YouTube · CheatCoders on YouTube

Most viewed

  1. Python Decorators Explained: From Simple Wrappers to Production Patterns233 views
  2. Claude Code vs Cursor vs GitHub Copilot: Choose by Workflow, and Set Up One Repo That Works With All Three105 views
  3. AI Agent Frameworks in 2025: LangGraph vs CrewAI vs AutoGen vs Raw API87 views
  4. Python String Methods: Every str Method With Real Production Examples72 views
  5. REST API Design Best Practices: The Patterns That Make APIs a Joy to Use64 views

Newly added

  1. n8n vs AI Agent: I Gave Both the Same Job (Local, Ollama)
  2. Local AI Video Generator: Build Your Own App (Free, Python, No GPU)
  3. AI Wrote Raw Binary Instead of Code. Here’s What Happened (WebAssembly)
  4. I Built a Browser That Only Runs Encrypted Binary (Ep 1)
  5. RAG From Scratch: Chunking, Embeddings, Reranking and Evaluation (Real Numbers)

Deep-dive PDF

Get the expanded guide for this post — extra diagrams-style checklists, failure modes, and a production walkthrough. Free when you subscribe to CheatCoders.

Already subscribed? or open the subscribe page.

Discover more from CheatCoders

Subscribe to get the latest posts sent to your email.

Comments

No comments yet. Why don’t you start the discussion?

Leave a comment

No account needed. Name and email are optional.