Lambda Event Source Mapping: Parallelization Factors That Match Sinks

Cranking SQS MaximumConcurrency or Kinesis parallelization factor feels like free lag reduction until DynamoDB starts returning ProvisionedThroughputExceeded and every retry multiplies load. Senior teams size event source mapping (ESM) parallelism against the sink, not the queue depth.

⚡ TL;DR: Model sink RPS/WCU first; set SQS max concurrency and Kinesis parallelization so peak fan-out ≤ sink capacity × safety factor; use partial batch failure + bisect where available; alarm on iterator age and sink throttle metrics together. Pair with Lambda SQS Partial Batch Failures and AWS Lambda Best Practices.

Size from the sink upward

required_concurrency ≈ ceil(peak_ingest_rps / records_per_invoke) 
sink_limit_rps       ≈ dynamo_wcu / writes_per_record   (or API quota)
safe_concurrency     ≈ floor(sink_limit_rps / rps_per_concurrent_invoke * 0.7)

Pick min(required, safe). Anything above safe buys lag reduction today and a throttle storm tomorrow.

// cdk/esm.ts
import * as lambda from "aws-cdk-lib/aws-lambda";
import * as lambdaEventSources from "aws-cdk-lib/aws-lambda-event-sources";
import * as sqs from "aws-cdk-lib/aws-sqs";

const queue = sqs.Queue.fromQueueArn(this, "Ingest", queueArn);
const fn = new lambda.Function(this, "Writer", { /* ... */ reservedConcurrentExecutions: 50 });

fn.addEventSource(new lambdaEventSources.SqsEventSource(queue, {
  batchSize: 10,
  maxBatchingWindow: Duration.seconds(1),
  reportBatchItemFailures: true,
  // ✅ Cap fan-out to match DynamoDB writer budget (~35 concurrent * 10 rec ≈ sink)
  maxConcurrency: 35,
}));

Kinesis parallelization factor is not free

For Kinesis/DynamoDB Streams, parallelizationFactor multiplies concurrent invokes per shard. Factor 10 on 100 shards is 1000 concurrent Lambdas — each must fit reserved concurrency and sink quotas.

# CloudFormation fragment
EventSourceMapping:
  Type: AWS::Lambda::EventSourceMapping
  Properties:
    FunctionName: !Ref WriterFn
    EventSourceArn: !GetAtt Stream.Arn
    StartingPosition: LATEST
    BatchSize: 100
    ParallelizationFactor: 2   # ✅ start low; raise only with sink headroom
    BisectBatchOnFunctionError: true
    FunctionResponseTypes:
      - ReportBatchItemFailures

❌ Setting parallelization to 10 “because lag is high” while the handler does a hot-partition Dynamo write per record.

Couple ESM alarms to sink throttles

# scripts/size_esm.py — illustrative planner
def suggest_sqs_max_concurrency(peak_rps: float, batch: int, sink_rps: float, safety=0.7) -> int:
    need = max(1, int((peak_rps + batch - 1) // batch))
    safe = max(1, int(sink_rps * safety / max(peak_rps / need, 1e-9)))
    # rps_per_invoke ≈ peak_rps/need; simplify:
    safe = max(1, int((sink_rps * safety) / (peak_rps / need)))
    return min(need, safe)

Closing checklist

✅ Dos
– ✅ Derive concurrency from sink WCU/RPS/quotas
– ✅ Enable partial batch failure reporting
– ✅ Reserve concurrency so ESM cannot starve the rest of the account
– ✅ Load-test with production-shaped keys (hot partitions)
– ✅ Dual-alarm: lag + sink throttles

❌ Don’ts
– ❌ Don’t max out parallelization factor blindly
– ❌ Don’t ignore reserved concurrency when ESM scales
– ❌ Don’t treat queue depth as the only scaling signal
– ❌ Don’t share one over-parallel consumer across unrelated sinks

Related reading

• Lambda SQS Partial Batch Failures
• Lambda Kinesis Bisect on Error
• Lambda Timeouts, Retries, and DLQs
• Lambda Concurrency: Provisioned, SnapStart, Reserved

Last updated on September 11, 2026