Lambda Kinesis Bisect on Error: Isolate Bad Records Without Lag

One corrupt Kinesis record can pin a shard’s iterator age for hours if the function fails the whole batch. Bisect-on-error splits the batch until the poison is isolated — but without partial batch response discipline and a DLQ/on-failure path, you still stall or drop data blindly.

⚡ TL;DR: Enable BisectBatchOnFunctionError + ReportBatchItemFailures; keep handlers idempotent; set MaximumRetryAttempts and OnFailure to SQS/SNS; alarm on iterator age per shard. Pair with Lambda SQS Partial Batch Failures and Lambda Event Source Mapping Parallelization.

Configure the mapping

Type: AWS::Lambda::EventSourceMapping
Properties:
  FunctionName: !Ref StreamConsumer
  EventSourceArn: !GetAtt OrdersStream.Arn
  StartingPosition: LATEST
  BatchSize: 100
  MaximumBatchingWindowInSeconds: 5
  ParallelizationFactor: 1
  BisectBatchOnFunctionError: true
  MaximumRetryAttempts: 3
  FunctionResponseTypes:
    - ReportBatchItemFailures
  DestinationConfig:
    OnFailure:
      Destination: !GetAtt StreamFailures.Arn

Handler with sequence checkpoints

// kinesisPartial.ts
import type { KinesisStreamEvent, KinesisStreamBatchResponse } from "aws-lambda";

export async function handler(event: KinesisStreamEvent): Promise<KinesisStreamBatchResponse> {
  const failures: { itemIdentifier: string }[] = [];

  for (const rec of event.Records) {
    try {
      const payload = Buffer.from(rec.kinesis.data, "base64").toString("utf8");
      await process(JSON.parse(payload), rec.kinesis.sequenceNumber);
    } catch (err) {
      console.error(JSON.stringify({
        msg: "kinesis_record_failed",
        seq: rec.kinesis.sequenceNumber,
        err: String(err),
      }));
      // ✅ itemIdentifier must be the sequence number
      failures.push({ itemIdentifier: rec.kinesis.sequenceNumber });
    }
  }
  return { batchItemFailures: failures };
}

Bisect behavior: on hard failure (uncaught), Lambda splits the batch and retries halves. With partial responses, only reported sequence numbers retry — faster isolation when you classify errors correctly.

Avoid multi-hour lag

// ❌ Failing the whole handler on JSON.parse of one record without partial response
export async function handler(event: KinesisStreamEvent) {
  for (const rec of event.Records) {
    JSON.parse(Buffer.from(rec.kinesis.data, "base64").toString("utf8")); // one bad record → full batch fail
  }
}

Alarm:

aws cloudwatch put-metric-alarm --alarm-name kinesis-iterator-age \
  --namespace AWS/Lambda --metric-name IteratorAge \
  --dimensions Name=FunctionName,Value=stream-consumer \
  --statistic Maximum --period 60 --evaluation-periods 5 \
  --threshold 60000 --comparison-operator GreaterThanThreshold

Closing checklist

✅ Dos
– ✅ Bisect + ReportBatchItemFailures together
– ✅ On-failure destination for exhausted retries
– ✅ Idempotent processing by sequence/business key
– ✅ Alarm iterator age and failure destination depth
– ✅ Keep batch window/size matched to sink latency

❌ Don’ts
– ❌ Don’t enable bisect without a failure destination
– ❌ Don’t raise parallelization to “fix” poison lag
– ❌ Don’t use shard iterators manually inside Lambda when ESM exists
– ❌ Don’t ignore schema evolution that creates sudden poison rates

Related reading

• Lambda SQS Partial Batch Failures
• Lambda Event Source Mapping Parallelization
• Lambda Destinations
• Lambda Timeouts, Retries, and DLQs

Last updated on September 11, 2026