Lambda SQS Partial Batch Failures: Poison Messages Without Infinite Redrive

Lambda SQS Partial Batch Failures: Poison Messages Without Infinite Redrive

With classic SQS→Lambda, throwing on one bad record redrives the entire batch. Healthy messages get duplicate side effects; poison messages loop until the DLQ — or forever if you misconfigure maxReceiveCount. Partial batch failure reporting fixes the contract: return only the failing message IDs.

⚡ TL;DR: Enable ReportBatchItemFailures; process records independently; return { batchItemFailures: [{ itemIdentifier }] }; classify poison vs transient; DLQ after N receives; idempotent handlers. Pair with Lambda Event Source Mapping Parallelization and Lambda Timeouts & DLQs.

Handler contract (Node 20)

// sqsPartial.ts
import type { SQSEvent, SQSBatchResponse, SQSRecord } from "aws-lambda";

export async function handler(event: SQSEvent): Promise<SQSBatchResponse> {
  const failures: { itemIdentifier: string }[] = [];

  await Promise.all(
    event.Records.map(async (rec) => {
      try {
        await processOne(rec);
      } catch (err) {
        console.error(JSON.stringify({
          msg: "record_failed",
          messageId: rec.messageId,
          err: String(err),
        }));
        failures.push({ itemIdentifier: rec.messageId });
      }
    })
  );

  // ✅ Only failing IDs — successes are deleted by Lambda/SQS
  return { batchItemFailures: failures };
}

async function processOne(rec: SQSRecord) {
  const body = JSON.parse(rec.body);
  if (!body.orderId) {
    // Poison: validation error — still report failure so it counts toward redrive,
    // or route to a quarantine queue explicitly if you prefer not to retry.
    throw new Error("poison_missing_orderId");
  }
  await idempotentWrite(body); // must tolerate duplicates
}

IaC switch

fn.addEventSource(new lambdaEventSources.SqsEventSource(queue, {
  batchSize: 10,
  reportBatchItemFailures: true, // ✅ required
  maxConcurrency: 40,
}));
# Queue redrive — poison lands here after N receives
RedrivePolicy:
  deadLetterTargetArn: !GetAtt Dlq.Arn
  maxReceiveCount: 5

Poison vs transient

Error class Action
Schema invalid / forever-fail Fail item → DLQ after maxReceive; alert
Downstream 429 / timeout Fail item → retry; backoff via visibility
Bug in code Fail item → fix + replay from DLQ
Handler timeout mid-batch Untouched records may redrive — keep batches small
// ❌ Anti-pattern: catch-all swallow
try {
  await processOne(rec);
} catch {
  // silent — message deleted, data lost
}

Closing checklist

✅ Dos
– ✅ ReportBatchItemFailures + return messageIds only
– ✅ Idempotent writes keyed by business id
– ✅ DLQ with alarm on ApproximateNumberOfMessagesVisible
– ✅ Separate validation poisons from transient throttles in metrics
– ✅ Keep batch size modest relative to timeout

❌ Don’ts
– ❌ Don’t throw at the top level for a single bad record
– ❌ Don’t swallow errors (acks poison as success)
– ❌ Don’t set maxReceiveCount to absurd values without a DLQ
– ❌ Don’t process the batch as one transaction unless you mean it

Related reading

Last updated on September 11, 2026


Discover more from CheatCoders

Subscribe to get the latest posts sent to your email.

1 Comment

Leave a Reply