8 views
Kafka consumer lag is a feature until Lambda parallelization turns it into a thundering herd against DynamoDB, RDS, or a partner API. Backpressure means: define lag SLOs per topic, cap concurrent invokes, and shed or slow consumption when sink error rates or latency burn the budget—not when the pager already screams.
⚡ TL;DR: Set lag SLO (e.g., p99 lag < 30s); bound event-source parallelization / reserved concurrency; propagate sink 429/5xx into slower polls or fewer shards processed; never “fix” lag by maxing concurrency into a choking DB. Related: Lambda Event Source Mapping: Parallelization Factors That Match Sinks, Lambda Reserved Concurrency: Bulkheads, Lambda Response Streaming: Backpressure.
Kafka to Lambda Backpressure: production guidance
text
Lag growing while sink p95 is healthy may mean under-scaled consumers. Lag flat while sink 5xx climbs means you need less concurrency.
SLO: consumer_lag_seconds{topic="orders"} p99 < 30 over 5m
Error budget: burn alerts at 2x / 5x consumption rate
Cap the herd
For MSK/Kafka → Lambda (or Kafka → SQS → Lambda), set reserved concurrency to what the sink sustains in soak tests. Parallelization factor / per-partition concurrency must match write capacity.yaml
# SAM sketch — concurrency bulkhead
OrderConsumer:
Type: AWS::Serverless::Function
Properties:
ReservedConcurrentExecutions: 50
Events:
Kafka:
Type: MSK
Properties:
Stream: !Ref OrdersCluster
Topics:
- orders
BatchSize: 100
python
# ❌ Unbounded: scale Lambda to wipe lag at all costs
# ReservedConcurrentExecutions omitted + high parallelization
Adaptive throttle signal
Expose sink health to the consumer path:python
Prefer partial batch failure where supported so healthy partitions proceed. For classic Kafka consumers on ECS, lower # consumer/handler.py
class SinkBusy(Exception):
pass
def handler(event, context):
if sink_token_bucket.try_take(len(event["records"])) is False:
# Return failures so batch retries later — or pause partition
raise SinkBusy("dynamo_throttle")
process_batch(event["records"])
max.poll.records and pause partitions when circuit breakers open.
Protecting the sink
| Signal | Action |
|---|---|
| Sink throttle (429) | Cut concurrency 50%, alarm |
| Sink p95 > SLO | Pause non-critical topics first |
| Lag > SLO but sink OK | Scale consumers carefully |
| Poison messages | Quarantine (bisect patterns) |
Closing checklist
✅ Dos – ✅ Publish lag histograms and SLO burn alerts – ✅ Soak-test sink capacity; set reserved concurrency from evidence – ✅ Circuit-break when sink throttles – ✅ Prefer partial failures over all-or-nothing batch retry – ✅ Separate critical vs deferred topics ❌ Don’ts – ❌ Don’t max parallelization to “clear lag” into RDS – ❌ Don’t ignore 429s as blips – ❌ Don’t use one shared concurrency pool for all tenants/topics – ❌ Don’t alert only on absolute lag without rate of change – ❌ Don’t skip DLQ/quarantine for poison payloadsRelated reading
- Lambda Event Source Mapping: Parallelization Factors That Match Sinks
- Lambda Reserved Concurrency: Bulkheads That Protect Tenant Workloads
- Lambda Response Streaming: Backpressure When Clients Stall Mid-Transfer
- Lambda Kinesis Bisect on Error: Isolate Bad Records Without Lag
Watch: 100 Days of AI on YouTube
Short videos from CheatCoders, one AI engineering topic per day.
Most viewed
- Python Decorators Explained: From Simple Wrappers to Production Patterns
- Claude Code vs Cursor vs GitHub Copilot: Choose by Workflow, and Set Up One Repo That Works With All Three
- AI Agent Frameworks in 2025: LangGraph vs CrewAI vs AutoGen vs Raw API
- Python String Methods: Every str Method With Real Production Examples
- Css Animation Performance: Performance-Optimized Motion for Modern UIs
Newly added
- AWS Systems Manager Parameter Store: Hierarchical Runtime Config for Coding Agents
- Amazon Cognito Identity Pools: Federate Coding-Agent Tool Callers Without Long-Lived Keys
- Amazon Athena for Coding Agents: Query Tool Audit Logs and Eval Traces on S3 Without a Warehouse
- AWS Batch for Coding Agents: Parallel Overnight PR Reviews and Long-Running Jobs Without Lambda Timeouts
- Amazon Bedrock Prompt Management: Version and A/B Test Coding-Agent System Prompts
Deep-dive PDF
Get the expanded guide for this post — extra diagrams-style checklists, failure modes, and a production walkthrough. Free when you subscribe to CheatCoders.
Already subscribed? or open the subscribe page.
Discover more from CheatCoders
Subscribe to get the latest posts sent to your email.
