Lambda Timeouts, Retries, and DLQs: Idempotent Failure Handling

The ugliest Lambda incidents are not “function timed out.” They are timeouts plus retries plus side effects: a payment captured twice, a fulfillment message published three times, a DLQ nobody owns. Node handlers that await a 29s HTTP call inside a 30s function timeout are a coin flip; async event sources that retry forever without idempotency keys are a slow-motion outage. This is the production shape: budgets at every hop, retries that are safe by construction, and DLQs with a replay runbook.

⚡ TL;DR: Set Lambda timeout to work budget + jitter cushion, not 900s “for safety.” Cap downstream HTTP/SDK timeouts below the function timeout. For SQS/SNS/EventBridge/Streams, understand the retry contract and make handlers idempotent with DynamoDB conditional writes or idempotency keys (Powertools). Wire DLQs (or failed-event destinations) and alert on depth — a silent DLQ is a deferred SEV-1. Illustrative targets: API handlers 3–10s; queue workers 30–60s with partial batch failure; poison messages isolated within minutes, not days.

Timeout budgets: function clock vs dependency clock

Lambda kills the isolate when the configured timeout hits. Your AWS SDK client may still be waiting unless you set its timeout lower. Two clocks, one process.

// handler.ts — explicit budgets beat hope
import { DynamoDBClient } from "@aws-sdk/client-dynamodb";
import { NodeHttpHandler } from "@smithy/node-http-handler";

// Function timeout in console/Terraform: 8s for this API path (illustrative)
const ddb = new DynamoDBClient({
  requestHandler: new NodeHttpHandler({
    connectionTimeout: 1_000,
    requestTimeout: 2_500, // ✅ well under 8s function timeout
  }),
  maxAttempts: 2, // ✅ finite; don’t stack SDK retries × async retries blindly
});

export async function handler(event: { orderId: string }) {
  const ac = new AbortController();
  const kill = setTimeout(() => ac.abort(), 6_000); // leave ~2s for cleanup/logging
  try {
    // pass ac.signal into fetch / ORM calls
    const res = await fetch(process.env.PRICING_URL! + "/" + event.orderId, {
      signal: ac.signal,
    });
    if (!res.ok) throw new Error(`pricing_${res.status}`);
    return { ok: true };
  } finally {
    clearTimeout(kill);
  }
}

✅ Function timeout ≥ sum of worst-case dependency budgets + logging cushion.
❌ Function timeout 900s with default SDK retries — you will pay for hung sockets and duplicate work.

For API Gateway + Lambda, remember the integration timeout ceiling (historically ~29s for REST; check current HTTP API limits). A 60s Lambda behind API Gateway does not give you a 60s client experience.

Retries differ by event source — learn the contract

// SQS partial batch failure — Node 20 / AWS Lambda
import type { SQSBatchResponse, SQSEvent } from "aws-lambda";

export async function handler(event: SQSEvent): Promise<SQSBatchResponse> {
  const batchItemFailures: { itemIdentifier: string }[] = [];

  await Promise.all(
    event.Records.map(async (record) => {
      try {
        await processOnce(record.body, record.messageId);
      } catch (err) {
        console.error("record_failed", { id: record.messageId, err: String(err) });
        // ✅ only failing ids retry; successes delete
        batchItemFailures.push({ itemIdentifier: record.messageId });
      }
    })
  );

  return { batchItemFailures };
}

// ❌ throw on first failure with ReportBatchItemFailures off → whole batch retries → dupes

Enable FunctionResponseTypes = ReportBatchItemFailures on the event source mapping. Pair with a redrive policy:

resource "aws_sqs_queue" "orders" {
  name                      = "orders"
  visibility_timeout_seconds = 60 # ≥ 6× Lambda timeout is a common rule of thumb
  redrive_policy = jsonencode({
    deadLetterTargetArn = aws_sqs_queue.orders_dlq.arn
    maxReceiveCount     = 3
  })
}

resource "aws_sqs_queue" "orders_dlq" {
  name = "orders-dlq"
}

Visibility timeout too low → parallel double-processing. Too high → slow recovery. Measure.

Idempotency: the only safe retry story

Retries are mandatory in distributed systems. Side effects must tolerate them.

// idempotency with DynamoDB conditional put — unfair-advantage minimal version
import { DynamoDBClient } from "@aws-sdk/client-dynamodb";
import {
  DynamoDBDocumentClient,
  PutCommand,
  GetCommand,
} from "@aws-sdk/lib-dynamodb";

const doc = DynamoDBDocumentClient.from(new DynamoDBClient({}));
const TABLE = process.env.IDEMPOTENCY_TABLE!;

export async function processOnce(body: string, messageId: string) {
  const payload = JSON.parse(body) as { orderId: string; amountCents: number };
  const pk = `ORDER#${payload.orderId}`;

  try {
    await doc.send(
      new PutCommand({
        TableName: TABLE,
        Item: {
          pk,
          status: "IN_FLIGHT",
          messageId,
          ttl: Math.floor(Date.now() / 1000) + 86400,
        },
        ConditionExpression: "attribute_not_exists(pk) OR #s = :failed",
        ExpressionAttributeNames: { "#s": "status" },
        ExpressionAttributeValues: { ":failed": "FAILED" },
      })
    );
  } catch (e: any) {
    if (e?.name === "ConditionalCheckFailedException") {
      const existing = await doc.send(new GetCommand({ TableName: TABLE, Key: { pk } }));
      if (existing.Item?.status === "COMPLETED") {
        // ✅ safe replay
        return existing.Item.result;
      }
      throw e; // still in flight — let SQS retry with backoff
    }
    throw e;
  }

  const result = await chargeCard(payload); // side effect
  await doc.send(
    new PutCommand({
      TableName: TABLE,
      Item: { pk, status: "COMPLETED", result, messageId, ttl: Math.floor(Date.now() / 1000) + 86400 },
    })
  );
  return result;
}

Or use Lambda Powertools Idempotency (@aws-lambda-powertools/idempotency) with a DynamoDB persistence store — same idea, less glue. Key on a business id (orderId), not only messageId, so SNS redeliveries and client retries collapse correctly.

✅ Idempotency key = business intent.
❌ “It’s OK, SQS is exactly-once” — it is not.

DLQs and failure destinations you actually operate

A DLQ without an alarm is a trash folder. A DLQ without a replay path is how you lose orders forever.

// CloudWatch alarm sketch (conceptual)
// ApproximateNumberOfMessagesVisible > 0 for 5 minutes on orders-dlq → page

// Replay tool — intentional, rate-limited, idempotent handler still required
import { SQSClient, ReceiveMessageCommand, SendMessageCommand, DeleteMessageCommand } from "@aws-sdk/client-sqs";

const sqs = new SQSClient({});

export async function replayDlq(dlqUrl: string, mainUrl: string, max = 10) {
  const got = await sqs.send(new ReceiveMessageCommand({
    QueueUrl: dlqUrl,
    MaxNumberOfMessages: max,
    VisibilityTimeout: 30,
  }));
  for (const m of got.Messages ?? []) {
    await sqs.send(new SendMessageCommand({ QueueUrl: mainUrl, MessageBody: m.Body! }));
    await sqs.send(new DeleteMessageCommand({ QueueUrl: dlqUrl, ReceiptHandle: m.ReceiptHandle! }));
  }
}

For asynchronous Lambda invokes (SNS, S3, EventBridge→Lambda), configure OnFailure destinations to SQS/SNS instead of relying on luck:

aws lambda put-function-event-invoke-config \
  --function-name fulfill-order \
  --maximum-retry-attempts 2 \
  --destination-config '{"OnFailure":{"Destination":"arn:aws:sqs:REGION:ACCT:fulfill-dlq"}}'

Log the payload hash, business key, and error code into the DLQ message attributes so humans can triage without decrypting archaeology.

Observability: timeout vs error vs throttle

Emit explicit outcomes so dashboards don’t lie:

import { Logger } from "@aws-lambda-powertools/logger";
import { Metrics, MetricUnit } from "@aws-lambda-powertools/metrics";

const logger = new Logger({ serviceName: "orders" });
const metrics = new Metrics({ namespace: "CheatCoders/Orders", serviceName: "orders" });

export async function guarded<T>(name: string, fn: () => Promise<T>): Promise<T> {
  const start = Date.now();
  try {
    const out = await fn();
    metrics.addMetric(`${name}Success`, MetricUnit.Count, 1);
    return out;
  } catch (err: any) {
    const timedOut = err?.name === "AbortError" || /Timeout/i.test(String(err));
    metrics.addMetric(timedOut ? `${name}Timeout` : `${name}Error`, MetricUnit.Count, 1);
    logger.error("downstream_failed", { name, timedOut, err: String(err), ms: Date.now() - start });
    throw err;
  } finally {
    metrics.publishStoredMetrics();
  }
}

Pair timeouts with the error-handling discipline in Node.js Error Handling: Production Patterns and Monitoring and queue design in Event-Driven Architecture: Decoupled Systems With Message Queues. Cold starts that burn half your timeout budget belong in Lambda Cold Starts on Node 20 (companion in this batch).

Closing checklist

✅ Dos
– ✅ Set SDK/HTTP timeouts below the function timeout; leave cushion for logs
– ✅ Enable partial batch failure on SQS; tune visibility ≥ Lambda timeout
– ✅ Idempotency keys on every mutating consumer
– ✅ DLQ + alarm + documented replay
– ✅ Metric timeout vs error vs throttle separately

❌ Don’ts
– ❌ Don’t default to 900s timeouts to “stop the noise”
– ❌ Don’t stack infinite SDK retries on top of SQS retries
– ❌ Don’t treat messageId alone as business idempotency
– ❌ Don’t ship a DLQ without an owner and a page
– ❌ Don’t replay DLQs into non-idempotent handlers

Related reading

• AWS Lambda Best Practices: Write Functions That Scale and Never Time Out
• Event-Driven Architecture: Decoupled Systems With Message Queues
• Node.js Error Handling: Production Patterns and Monitoring
• AWS DynamoDB: Advanced Patterns for Production at Scale
• Lambda Cold Starts on Node 20: Measure, Cut, and Keep Cutting
• Lambda Powertools for Node: Structured Logs That Survive On-Call

Last updated on September 10, 2026