8 views
This guide covers Lambda recursive loop detection with production-focused examples and trade-offs.
One miswired SNS → Lambda → SNS, or a “helpful” handler that re-invokes itself on every error, can exhaust regional concurrency and print a five-figure bill before anyone wakes up. AWS recursive loop detection is the seatbelt; your architecture and billing alarms are the brakes.
⚡ TL;DR: Prefer event designs that cannot fan into themselves; enable recursive loop detection; stamp idempotency + hop counters on every async invoke; alarm on RecursiveInvocationsDropped and unexpected spend. Pair with Lambda Destinations and Lambda Timeouts & DLQs.
How loops form — Lambda recursive loop detection
Classic patterns:- SQS → Lambda → same SQS (publish on failure without poison handling)
- SNS topic A → Lambda → publish to topic A
- Async self-invoke “retry” without a counter
- S3 event on bucket B → Lambda writes to bucket B under a watched prefix
typescript
// ❌ Accidental loop: error path republishes same message shape to same bus
await eventBridge.putEvents({
Entries: [{
EventBusName: "app",
Source: "orders",
DetailType: "OrderFailed",
Detail: JSON.stringify(event.detail), // re-enters the same rule
}],
});
typescript
// ✅ Hop counter + different detail type for remediation
const hops = Number(event.detail?.hops ?? 0);
if (hops >= 3) {
await sendToDestinations(event); // on-failure queue, not the bus
return;
}
await eventBridge.putEvents({
Entries: [{
EventBusName: "app",
Source: "orders.remediator",
DetailType: "OrderRemediationRequested",
Detail: JSON.stringify({ ...event.detail, hops: hops + 1 }),
}],
});
Turn on detection and watch the metric
Recursive loop detection drops invokes that look like cycles among supported AWS services and emitsRecursiveInvocationsDropped. Treat that metric as a sev page, not a curiosity.
bash
Also alarm on:
# Alarm sketch
aws cloudwatch put-metric-alarm \
--alarm-name lambda-recursive-dropped \
--metric-name RecursiveInvocationsDropped \
--namespace AWS/Lambda \
--statistic Sum \
--period 60 \
--evaluation-periods 1 \
--threshold 1 \
--comparison-operator GreaterThanOrEqualToThreshold \
--dimensions Name=FunctionName,Value=orders-consumer \
--alarm-actions arn:aws:sns:region:acct:oncall
UnreservedConcurrentExecutionapproaching account limit- Cost anomaly / daily Lambda spend budget
- SQS
NumberOfMessagesSentself-correlation with the consumer function
Defense in depth beyond AWS detection
| Control | Role |
|---|---|
| Hop / TTL attribute | App-level hard stop |
| Idempotency keys | Stop duplicate side effects |
| Reserved concurrency | Blast-radius cap |
| On-failure destinations | Escape hatch ≠ same bus |
| Separate remediator function | Break identity cycle |
python
# assert_no_self_invoke.py (CI check on IaC)
import json, sys
template = json.load(open(sys.argv[1]))
# naive: flag Lambda whose EventBridge rule target equals its own publish bus+source
# fail build if detected without explicit allow annotation
Closing checklist
✅ Dos– ✅ Design acyclic event graphs; document allowed remediations
– ✅ Enable recursive loop detection on async workflows
– ✅ Page on
RecursiveInvocationsDropped– ✅ Cap reserved concurrency on risky consumers
– ✅ Use destinations/DLQ instead of republishing the same event ❌ Don’ts
– ❌ Don’t async self-invoke without a hop budget
– ❌ Don’t write S3 notifications back to the watched prefix
– ❌ Don’t disable detection to “fix retries”
– ❌ Don’t rely on humans noticing the bill first
Related reading
- Lambda Destinations: Async Failure Routing
- Lambda Timeouts, Retries, and DLQs
- AWS Lambda Best Practices
- Lambda Event Source Mapping Parallelization
Related: Lambda Recursive Loop Protection: Stop Agent Self-Invokes From Burning Quotas · Lambda Invoke Modes: BUFFERED Versus RESPONSE_STREAM by Client Type · Day 43: Distillation for Edge and Lambda
Watch: 100 Days of AI on YouTube
Short videos from CheatCoders, one AI engineering topic per day.
Most viewed
- Python Decorators Explained: From Simple Wrappers to Production Patterns
- Claude Code vs Cursor vs GitHub Copilot: Choose by Workflow, and Set Up One Repo That Works With All Three
- AI Agent Frameworks in 2025: LangGraph vs CrewAI vs AutoGen vs Raw API
- Python String Methods: Every str Method With Real Production Examples
- Css Animation Performance: Performance-Optimized Motion for Modern UIs
Newly added
- AWS Systems Manager Parameter Store: Hierarchical Runtime Config for Coding Agents
- Amazon Cognito Identity Pools: Federate Coding-Agent Tool Callers Without Long-Lived Keys
- Amazon Athena for Coding Agents: Query Tool Audit Logs and Eval Traces on S3 Without a Warehouse
- AWS Batch for Coding Agents: Parallel Overnight PR Reviews and Long-Running Jobs Without Lambda Timeouts
- Amazon Bedrock Prompt Management: Version and A/B Test Coding-Agent System Prompts
Deep-dive PDF
Get the expanded guide for this post — extra diagrams-style checklists, failure modes, and a production walkthrough. Free when you subscribe to CheatCoders.
Already subscribed? or open the subscribe page.
Discover more from CheatCoders
Subscribe to get the latest posts sent to your email.
