6 views
This guide covers Lambda event source mapping with production-focused examples and trade-offs.
Cranking SQSMaximumConcurrency or Kinesis parallelization factor feels like free lag reduction until DynamoDB starts returning ProvisionedThroughputExceeded and every retry multiplies load. Senior teams size event source mapping (ESM) parallelism against the sink, not the queue depth.
⚡ TL;DR: Model sink RPS/WCU first; set SQS max concurrency and Kinesis parallelization so peak fan-out ≤ sink capacity × safety factor; use partial batch failure + bisect where available; alarm on iterator age and sink throttle metrics together. Pair with Lambda SQS Partial Batch Failures and AWS Lambda Best Practices.
Size from the sink upward — Lambda event source mapping
text
Pick required_concurrency ≈ ceil(peak_ingest_rps / records_per_invoke)
sink_limit_rps ≈ dynamo_wcu / writes_per_record (or API quota)
safe_concurrency ≈ floor(sink_limit_rps / rps_per_concurrent_invoke * 0.7)
min(required, safe). Anything above safe buys lag reduction today and a throttle storm tomorrow.
typescript
// cdk/esm.ts
import * as lambda from "aws-cdk-lib/aws-lambda";
import * as lambdaEventSources from "aws-cdk-lib/aws-lambda-event-sources";
import * as sqs from "aws-cdk-lib/aws-sqs";
const queue = sqs.Queue.fromQueueArn(this, "Ingest", queueArn);
const fn = new lambda.Function(this, "Writer", { /* ... */ reservedConcurrentExecutions: 50 });
fn.addEventSource(new lambdaEventSources.SqsEventSource(queue, {
batchSize: 10,
maxBatchingWindow: Duration.seconds(1),
reportBatchItemFailures: true,
// ✅ Cap fan-out to match DynamoDB writer budget (~35 concurrent * 10 rec ≈ sink)
maxConcurrency: 35,
}));
Kinesis parallelization factor is not free
For Kinesis/DynamoDB Streams,parallelizationFactor multiplies concurrent invokes per shard. Factor 10 on 100 shards is 1000 concurrent Lambdas — each must fit reserved concurrency and sink quotas.
yaml
❌ Setting parallelization to 10 “because lag is high” while the handler does a hot-partition Dynamo write per record.
# CloudFormation fragment
EventSourceMapping:
Type: AWS::Lambda::EventSourceMapping
Properties:
FunctionName: !Ref WriterFn
EventSourceArn: !GetAtt Stream.Arn
StartingPosition: LATEST
BatchSize: 100
ParallelizationFactor: 2 # ✅ start low; raise only with sink headroom
BisectBatchOnFunctionError: true
FunctionResponseTypes:
- ReportBatchItemFailures
Couple ESM alarms to sink throttles
| Signal | Meaning | Action |
|---|---|---|
SQS ApproximateAgeOfOldestMessage ↑ |
Consumer lag | Scale concurrency if sink OK |
Dynamo ThrottledRequests ↑ |
Over-parallel | Lower maxConcurrency |
Kinesis IteratorAge ↑ + no throttles |
Under-parallel / slow code | Raise factor carefully |
Lambda ConcurrentExecutions at reserved |
Cap hit | Revisit reserved vs ESM |
python
# scripts/size_esm.py — illustrative planner
def suggest_sqs_max_concurrency(peak_rps: float, batch: int, sink_rps: float, safety=0.7) -> int:
need = max(1, int((peak_rps + batch - 1) // batch))
safe = max(1, int(sink_rps * safety / max(peak_rps / need, 1e-9)))
# rps_per_invoke ≈ peak_rps/need; simplify:
safe = max(1, int((sink_rps * safety) / (peak_rps / need)))
return min(need, safe)
Closing checklist
✅ Dos– ✅ Derive concurrency from sink WCU/RPS/quotas
– ✅ Enable partial batch failure reporting
– ✅ Reserve concurrency so ESM cannot starve the rest of the account
– ✅ Load-test with production-shaped keys (hot partitions)
– ✅ Dual-alarm: lag + sink throttles ❌ Don’ts
– ❌ Don’t max out parallelization factor blindly
– ❌ Don’t ignore reserved concurrency when ESM scales
– ❌ Don’t treat queue depth as the only scaling signal
– ❌ Don’t share one over-parallel consumer across unrelated sinks
Related reading
- Lambda SQS Partial Batch Failures
- Lambda Kinesis Bisect on Error
- Lambda Timeouts, Retries, and DLQs
- Lambda Concurrency: Provisioned, SnapStart, Reserved
Related: Lambda Event Source Filters: Drop Noise Before Paying for Invokes · Kafka to Lambda Backpressure: Lag SLOs That Protect Downstream Sinks · Day 43: Distillation for Edge and Lambda
Watch: 100 Days of AI on YouTube
Short videos from CheatCoders, one AI engineering topic per day.
Most viewed
- Python Decorators Explained: From Simple Wrappers to Production Patterns
- Claude Code vs Cursor vs GitHub Copilot: Choose by Workflow, and Set Up One Repo That Works With All Three
- AI Agent Frameworks in 2025: LangGraph vs CrewAI vs AutoGen vs Raw API
- Python String Methods: Every str Method With Real Production Examples
- Css Animation Performance: Performance-Optimized Motion for Modern UIs
Newly added
- AWS Systems Manager Parameter Store: Hierarchical Runtime Config for Coding Agents
- Amazon Cognito Identity Pools: Federate Coding-Agent Tool Callers Without Long-Lived Keys
- Amazon Athena for Coding Agents: Query Tool Audit Logs and Eval Traces on S3 Without a Warehouse
- AWS Batch for Coding Agents: Parallel Overnight PR Reviews and Long-Running Jobs Without Lambda Timeouts
- Amazon Bedrock Prompt Management: Version and A/B Test Coding-Agent System Prompts
Deep-dive PDF
Get the expanded guide for this post — extra diagrams-style checklists, failure modes, and a production walkthrough. Free when you subscribe to CheatCoders.
Already subscribed? or open the subscribe page.
Discover more from CheatCoders
Subscribe to get the latest posts sent to your email.
