Exactly-Once Illusions: Design At-Least-Once Plus Truly Idempotent Handlers

Brokers love the phrase “exactly-once.” Production failure modes do not. Network timeouts after a commit, replay from DLQs, and dual writes across systems recreate duplicates no matter what the marketing slide claimed. Seniors assume at-least-once everywhere and make handlers truly idempotent with durable keys and side-effect fencing.

⚡ TL;DR: Treat every delivery as a potential duplicate. Persist an idempotency record before side effects (or in the same transaction). Make effects themselves keyable (refunds, emails, ledger posts). Pair with Idempotency Keys End-to-End and Lambda Powertools Idempotency.

Where “exactly-once” quietly becomes twice

1. Handler commits DB work, crashes before ACK → broker redelivers
2. ACK lost on the wire → redelivery of already-processed message
3. DLQ replay after a bugfix → intentional duplicates
4. Two consumers in a "exclusive" group during rebalance overlap
5. Upstream producer retries after 503 with a new broker message id

❌ Deduping only on broker offset — producer retries mint new offsets for the same business event.

Idempotency record pattern

// consume/idempotent.ts
export async function handleOnce(eventId: string, fn: () => Promise<void>) {
  try {
    await ddb.put({
      Item: { pk: `idemp#${eventId}`, sk: "v1", status: "in_progress", at: Date.now() },
      ConditionExpression: "attribute_not_exists(pk)",
    });
  } catch (e: any) {
    if (e.name === "ConditionalCheckFailedException") {
      const row = await ddb.get({ Key: { pk: `idemp#${eventId}`, sk: "v1" } });
      if (row.Item?.status === "done") return; // ✅ safe replay
      throw new Error("in_progress_elsewhere");
    }
    throw e;
  }
  try {
    await fn();
    await ddb.update({
      Key: { pk: `idemp#${eventId}`, sk: "v1" },
      UpdateExpression: "SET #s = :d",
      ExpressionAttributeNames: { "#s": "status" },
      ExpressionAttributeValues: { ":d": "done" },
    });
  } catch (err) {
    await ddb.delete({ Key: { pk: `idemp#${eventId}`, sk: "v1" } }); // allow retry
    throw err;
  }
}

Prefer transactional outbox: write business row + idempotency marker atomically when the store allows.

Side effects must accept keys too

# effects/email.py
def send_receipt(event_id: str, to: str, body: str):
    try:
        table.put_item(
            Item={"pk": f"email#{event_id}", "to": to},
            ConditionExpression="attribute_not_exists(pk)",
        )
    except ClientError as e:
        if e.response["Error"]["Code"] == "ConditionalCheckFailedException":
            return  # ✅ already sent
        raise
    ses.send_email(to=to, body=body)

Testing duplicates on purpose

test("handler survives duplicate delivery", async () => {
  const evt = samplePaymentEvent();
  await handleOnce(evt.id, () => applyPayment(evt));
  await handleOnce(evt.id, () => applyPayment(evt)); // second time no-op
  expect(await ledger.count(evt.id)).toBe(1);
});

Wire this into CI alongside Hypothesis for Distributed Systems style reordering tests when you can.

Closing checklist

✅ Dos
– ✅ Assume at-least-once for every bus and queue
– ✅ Key on business event id, not broker offset alone
– ✅ Persist idempotency before external side effects
– ✅ Make downstream APIs accept idempotency keys
– ✅ Replay duplicates in automated tests

❌ Don’ts
– ❌ Don’t trust broker “EOS” as end-to-end exactly-once
– ❌ Don’t ACK before durable side effects complete
– ❌ Don’t use random UUIDs for effect keys on retry
– ❌ Don’t treat DLQ replay as a rare manual exception
– ❌ Don’t confuse batch de-dup with single-message safety

Related reading

• Idempotency Keys End-to-End: API Gateway Through Step Functions Safely
• Lambda Powertools Idempotency: DynamoDB Keys That Survive Retries
• Saga Compensations Under Partial Failure
• DynamoDB Streams Outbox

Last updated on September 11, 2026