Lambda Recursive Loop Detection: Break Accidental Invoke Storms Early

Lambda Recursive Loop Detection: Break Accidental Invoke Storms Early

One miswired SNS → Lambda → SNS, or a “helpful” handler that re-invokes itself on every error, can exhaust regional concurrency and print a five-figure bill before anyone wakes up. AWS recursive loop detection is the seatbelt; your architecture and billing alarms are the brakes.

⚡ TL;DR: Prefer event designs that cannot fan into themselves; enable recursive loop detection; stamp idempotency + hop counters on every async invoke; alarm on RecursiveInvocationsDropped and unexpected spend. Pair with Lambda Destinations and Lambda Timeouts & DLQs.

How loops form

Classic patterns:

  1. SQS → Lambda → same SQS (publish on failure without poison handling)
  2. SNS topic A → Lambda → publish to topic A
  3. Async self-invoke “retry” without a counter
  4. S3 event on bucket B → Lambda writes to bucket B under a watched prefix
// ❌ Accidental loop: error path republishes same message shape to same bus
await eventBridge.putEvents({
  Entries: [{
    EventBusName: "app",
    Source: "orders",
    DetailType: "OrderFailed",
    Detail: JSON.stringify(event.detail), // re-enters the same rule
  }],
});
// ✅ Hop counter + different detail type for remediation
const hops = Number(event.detail?.hops ?? 0);
if (hops >= 3) {
  await sendToDestinations(event); // on-failure queue, not the bus
  return;
}
await eventBridge.putEvents({
  Entries: [{
    EventBusName: "app",
    Source: "orders.remediator",
    DetailType: "OrderRemediationRequested",
    Detail: JSON.stringify({ ...event.detail, hops: hops + 1 }),
  }],
});

Turn on detection and watch the metric

Recursive loop detection drops invokes that look like cycles among supported AWS services and emits RecursiveInvocationsDropped. Treat that metric as a sev page, not a curiosity.

# Alarm sketch
aws cloudwatch put-metric-alarm \
  --alarm-name lambda-recursive-dropped \
  --metric-name RecursiveInvocationsDropped \
  --namespace AWS/Lambda \
  --statistic Sum \
  --period 60 \
  --evaluation-periods 1 \
  --threshold 1 \
  --comparison-operator GreaterThanOrEqualToThreshold \
  --dimensions Name=FunctionName,Value=orders-consumer \
  --alarm-actions arn:aws:sns:region:acct:oncall

Also alarm on:

  • UnreservedConcurrentExecution approaching account limit
  • Cost anomaly / daily Lambda spend budget
  • SQS NumberOfMessagesSent self-correlation with the consumer function

Defense in depth beyond AWS detection

Control Role
Hop / TTL attribute App-level hard stop
Idempotency keys Stop duplicate side effects
Reserved concurrency Blast-radius cap
On-failure destinations Escape hatch ≠ same bus
Separate remediator function Break identity cycle
# assert_no_self_invoke.py (CI check on IaC)
import json, sys
template = json.load(open(sys.argv[1]))
# naive: flag Lambda whose EventBridge rule target equals its own publish bus+source
# fail build if detected without explicit allow annotation

Closing checklist

✅ Dos
– ✅ Design acyclic event graphs; document allowed remediations
– ✅ Enable recursive loop detection on async workflows
– ✅ Page on RecursiveInvocationsDropped
– ✅ Cap reserved concurrency on risky consumers
– ✅ Use destinations/DLQ instead of republishing the same event

❌ Don’ts
– ❌ Don’t async self-invoke without a hop budget
– ❌ Don’t write S3 notifications back to the watched prefix
– ❌ Don’t disable detection to “fix retries”
– ❌ Don’t rely on humans noticing the bill first

Related reading

Last updated on September 11, 2026


Discover more from CheatCoders

Subscribe to get the latest posts sent to your email.

Comments

No comments yet. Why don’t you start the discussion?

Leave a Reply