Lambda Destinations: Async Failure Routing Beyond Plain SQS DLQs

A classic DLQ stores the original event — useful, but often not enough to replay safely. On-failure destinations add invoke context (request id, function arn, approximate timestamp, response payload) so remediator tools can route, dedupe, and replay without archaeology.

⚡ TL;DR: For async invokes, configure OnFailure to SQS or EventBridge with a versioned envelope schema; build a remediator that validates, dedupes by idempotency key, and re-invokes deliberately; keep DLQs as backup for stream sources. Pair with Lambda Recursive Loop Detection and Lambda Timeouts & DLQs.

Destinations vs DLQ

// cdk/destinations.ts
import * as lambda from "aws-cdk-lib/aws-lambda";
import * as destinations from "aws-cdk-lib/aws-lambda-destinations";
import * as sqs from "aws-cdk-lib/aws-sqs";
import * as events from "aws-cdk-lib/aws-events";

const failQ = new sqs.Queue(this, "FnOnFailure", {
  retentionPeriod: Duration.days(14),
});
const bus = events.EventBus.fromEventBusName(this, "Bus", "platform");

const fn = new lambda.Function(this, "AsyncWorker", {
  runtime: lambda.Runtime.NODEJS_20_X,
  handler: "index.handler",
  code: lambda.Code.fromAsset("dist"),
  retryAttempts: 2,
  onFailure: new destinations.SqsDestination(failQ),
  // alternatively: new destinations.EventBridgeDestination(bus)
});

Envelope schema for replay

Normalize destination payloads into your own schema the moment they land (EventBridge rule → remediator, or SQS consumer).

// types/failureEnvelope.ts
export type FailureEnvelopeV1 = {
  schema: "lambda.failure/v1";
  functionArn: string;
  requestId: string;
  timestamp: string;
  idempotencyKey: string; // from original event or hash
  payload: unknown;       // original async event
  error?: { type: string; message: string };
};

export function fromDestinationRecord(rec: any): FailureEnvelopeV1 {
  const req = rec.requestContext ?? rec.requestPayload ?? {};
  const payload = rec.requestPayload ?? rec;
  const key =
    payload?.idempotencyKey ??
    hashStable(payload);
  return {
    schema: "lambda.failure/v1",
    functionArn: rec.functionArn ?? req.functionArn,
    requestId: rec.requestContext?.requestId ?? "unknown",
    timestamp: new Date().toISOString(),
    idempotencyKey: key,
    payload,
    error: rec.responsePayload?.errorMessage
      ? { type: rec.responsePayload.errorType, message: rec.responsePayload.errorMessage }
      : undefined,
  };
}

Remediator pattern

// remediator.ts
export async function replay(env: FailureEnvelopeV1, mode: "dry-run" | "live") {
  if (await alreadySucceeded(env.idempotencyKey)) return { status: "skip" };
  if (mode === "dry-run") return { status: "would_replay", env };
  // ✅ Invoke updated alias deliberately — not the broken $LATEST by accident
  await lambda.send(new InvokeCommand({
    FunctionName: `${env.functionArn}:live`,
    InvocationType: "RequestResponse",
    Payload: Buffer.from(JSON.stringify(env.payload)),
  }));
  await markSucceeded(env.idempotencyKey);
  return { status: "replayed" };
}

❌ Auto-replaying every on-failure message into the same function version that just failed — without a circuit breaker.

Closing checklist

✅ Dos
– ✅ Use on-failure destinations for async invokes
– ✅ Version an envelope schema; index by idempotency key
– ✅ Separate remediator function + alias targeting
– ✅ Retain failure queues ≥ 14 days for incident windows
– ✅ Alarm on destination queue depth

❌ Don’ts
– ❌ Don’t confuse stream source DLQs with async destinations
– ❌ Don’t unbounded auto-replay
– ❌ Don’t drop requestId — you need it for X-Ray/logs
– ❌ Don’t publish failures back onto the originating bus without hop limits

Related reading

• Lambda Timeouts, Retries, and DLQs
• Lambda Recursive Loop Detection
• Lambda Alias Traffic Shifting
• AWS Lambda Best Practices

Last updated on September 11, 2026