One miswired SNS → Lambda → SNS, or a “helpful” handler that re-invokes itself on every error, can exhaust regional concurrency and print a five-figure bill before anyone wakes up. AWS recursive loop detection is the seatbelt; your architecture and billing alarms are the brakes.
⚡ TL;DR: Prefer event designs that cannot fan into themselves; enable recursive loop detection; stamp idempotency + hop counters on every async invoke; alarm on
RecursiveInvocationsDroppedand unexpected spend. Pair with Lambda Destinations and Lambda Timeouts & DLQs.
How loops form
Classic patterns:
- SQS → Lambda → same SQS (publish on failure without poison handling)
- SNS topic A → Lambda → publish to topic A
- Async self-invoke “retry” without a counter
- S3 event on bucket B → Lambda writes to bucket B under a watched prefix
// ❌ Accidental loop: error path republishes same message shape to same bus
await eventBridge.putEvents({
Entries: [{
EventBusName: "app",
Source: "orders",
DetailType: "OrderFailed",
Detail: JSON.stringify(event.detail), // re-enters the same rule
}],
});
// ✅ Hop counter + different detail type for remediation
const hops = Number(event.detail?.hops ?? 0);
if (hops >= 3) {
await sendToDestinations(event); // on-failure queue, not the bus
return;
}
await eventBridge.putEvents({
Entries: [{
EventBusName: "app",
Source: "orders.remediator",
DetailType: "OrderRemediationRequested",
Detail: JSON.stringify({ ...event.detail, hops: hops + 1 }),
}],
});
Turn on detection and watch the metric
Recursive loop detection drops invokes that look like cycles among supported AWS services and emits RecursiveInvocationsDropped. Treat that metric as a sev page, not a curiosity.
# Alarm sketch
aws cloudwatch put-metric-alarm \
--alarm-name lambda-recursive-dropped \
--metric-name RecursiveInvocationsDropped \
--namespace AWS/Lambda \
--statistic Sum \
--period 60 \
--evaluation-periods 1 \
--threshold 1 \
--comparison-operator GreaterThanOrEqualToThreshold \
--dimensions Name=FunctionName,Value=orders-consumer \
--alarm-actions arn:aws:sns:region:acct:oncall
Also alarm on:
UnreservedConcurrentExecutionapproaching account limit- Cost anomaly / daily Lambda spend budget
- SQS
NumberOfMessagesSentself-correlation with the consumer function
Defense in depth beyond AWS detection
| Control | Role |
|---|---|
| Hop / TTL attribute | App-level hard stop |
| Idempotency keys | Stop duplicate side effects |
| Reserved concurrency | Blast-radius cap |
| On-failure destinations | Escape hatch ≠ same bus |
| Separate remediator function | Break identity cycle |
# assert_no_self_invoke.py (CI check on IaC)
import json, sys
template = json.load(open(sys.argv[1]))
# naive: flag Lambda whose EventBridge rule target equals its own publish bus+source
# fail build if detected without explicit allow annotation
Closing checklist
✅ Dos
– ✅ Design acyclic event graphs; document allowed remediations
– ✅ Enable recursive loop detection on async workflows
– ✅ Page on RecursiveInvocationsDropped
– ✅ Cap reserved concurrency on risky consumers
– ✅ Use destinations/DLQ instead of republishing the same event
❌ Don’ts
– ❌ Don’t async self-invoke without a hop budget
– ❌ Don’t write S3 notifications back to the watched prefix
– ❌ Don’t disable detection to “fix retries”
– ❌ Don’t rely on humans noticing the bill first
Related reading
- Lambda Destinations: Async Failure Routing
- Lambda Timeouts, Retries, and DLQs
- AWS Lambda Best Practices
- Lambda Event Source Mapping Parallelization
Last updated on September 11, 2026
Discover more from CheatCoders
Subscribe to get the latest posts sent to your email.
