Lambda Destinations: Route Failed Agent Tool Invokes Without Silent Drops

This guide focuses on Lambda Destinations for production systems, with practical trade-offs for reliability, security, and cost.
Your coding agent fires lambda:InvokeFunction with InvocationType: Event so a lint sandbox or patch applicator runs in the background. The model already moved on — it thinks the tool “started.” Three retries later the worker dies on a permissions error and nothing tells the agent, the user, or your on-call. That is the silent-drop tax of async Invoke without Lambda destinations. OnFailure / OnSuccess destinations (SQS, SNS, EventBridge, or another Lambda) turn vanished work into recoverable events.

⚡ TL;DR: Async agent tool invokes need OnFailure (and often OnSuccess) destinations plus a DLQ mindset. Sync Invoke surfaces exceptions to the caller; Event invoke does not. Wire destinations like you wire SQS FIFO ordered agent queues, keep recursion budgets from Recursive Loop Protection, and resume UX over API Gateway WebSockets. Never fire-and-forget a mutating tool without a failure path.

Lambda Destinations: production guidance

Agents love Event invoke because it decouples latency. They also love to forget that 202 means accepted, not succeeded.

// ❌ Fire-and-forget tool — failures evaporate
import { LambdaClient, InvokeCommand } from "@aws-sdk/client-lambda";

const lambda = new LambdaClient({});

export async function runSandboxTool(input: {
  sessionId: string;
  patch: string;
}) {
  await lambda.send(
    new InvokeCommand({
      FunctionName: process.env.SANDBOX_FN!,
      InvocationType: "Event",
      Payload: Buffer.from(JSON.stringify(input)),
    })
  );
  // Model hears "started" — if sandbox crashes, nobody knows
  return { status: "started" };
}

Configure OnFailure and OnSuccess

Destinations attach to the function’s async invoke config, not to each Invoke call from the SDK (you can also override per-invoke with DestinationConfig in some paths — prefer the function default so every caller inherits safety).

// ✅ CDK — destinations for agent worker
import * as lambda from "aws-cdk-lib/aws-lambda";
import * as destinations from "aws-cdk-lib/aws-lambda-destinations";
import * as sqs from "aws-cdk-lib/aws-sqs";
import * as events from "aws-cdk-lib/aws-events";
import * as cdk from "aws-cdk-lib";

const failureQ = new sqs.Queue(this, "AgentToolFailureQ", {
  retentionPeriod: cdk.Duration.days(14),
});
const bus = new events.EventBus(this, "AgentBus");

const sandbox = new lambda.Function(this, "SandboxWorker", {
  // runtime, handler, ...
});

sandbox.configureAsyncInvoke({
  maxEventAge: cdk.Duration.hours(2),
  retryAttempts: 1, // ✅ agents: low retries + destination > thrash
  onFailure: new destinations.SqsDestination(failureQ),
  onSuccess: new destinations.EventBridgeDestination(bus),
});

OnFailure payload includes requestContext, responseContext, and the original event — enough to reconstruct sessionId and notify the IDE over WebSockets. OnSuccess is useful when the agent needs a completion signal without polling CloudWatch Logs.

// ✅ Failure consumer — notify session + park for replay
import type { SQSHandler } from "aws-lambda";
import {
  ApiGatewayManagementApiClient,
  PostToConnectionCommand,
} from "@aws-sdk/client-apigatewaymanagementapi";

const apigw = new ApiGatewayManagementApiClient({
  endpoint: process.env.WS_CALLBACK_URL!,
});

export const onToolFailure: SQSHandler = async (event) => {
  for (const rec of event.Records) {
    const dest = JSON.parse(rec.body);
    // Lambda destination envelope
    const original = dest.requestPayload ?? dest;
    const sessionId = original.sessionId as string | undefined;
    const connectionId = original.connectionId as string | undefined;
    const err =
      dest.responsePayload?.errorMessage ??
      dest.responseContext?.statusCode ??
      "tool_failed";

    if (connectionId) {
      try {
        await apigw.send(
          new PostToConnectionCommand({
            ConnectionId: connectionId,
            Data: Buffer.from(
              JSON.stringify({
                type: "tool_failed",
                sessionId,
                error: String(err),
              })
            ),
          })
        );
      } catch {
        // stale connection — session table TTL will clean up
      }
    }
    // persist for operator replay / idempotent redrive
  }
};

Why agents need destinations more than cron jobs

1. Fire-and-forget tools — model continues the turn; user never sees stderr
2. Partial graphs — parent agent invoked three workers; one died; plan assumes all three OK
3. Permission flaps — IAM eventual consistency → first tries fail, destination captures the corpse
4. Recursive / fan-out — without destinations, dropped recursive invokes look like “progress”

Prefer SQS as the primary transport for mutating tools (ordered FIFO + idempotency) and use Event invoke destinations as a belt-and-suspenders for the remaining async paths. Destinations are not a substitute for a proper job queue when you need exactly-once applies.

Destinations vs DLQ vs dead-letter on the queue

• Lambda async DLQ (legacy): SNS/SQS for failures after retries — destinations are the modern, richer replacement (success + failure, more targets).
• SQS redrive DLQ: for queue-triggered workers — still required; destinations do not replace queue DLQs.
• OnFailure → SQS: your operational inbox for Event-invoke tools.

// ✅ Per-invoke DestinationConfig when the default is not enough
await lambda.send(
  new InvokeCommand({
    FunctionName: process.env.SANDBOX_FN!,
    InvocationType: "Event",
    Payload: Buffer.from(JSON.stringify(input)),
    // SDK v3: DestinationConfig supported on Invoke for async
    DestinationConfig: {
      OnFailure: { Destination: process.env.FAILURE_QUEUE_ARN! },
      OnSuccess: { Destination: process.env.SUCCESS_TOPIC_ARN! },
    },
  })
);

IAM: the invoking principal needs lambda:InvokeFunction; the worker execution role needs sqs:SendMessage / sns:Publish / events:PutEvents on the destination ARN. Missing destination IAM is a common “I configured it but still silent” bug — check CloudWatch DestinationDeliveryFailures.

Patterns that still drop work

Checklist

• [ ] Every agent worker that accepts Event invoke has OnFailure → SQS (or SNS)
• [ ] OnSuccess routed when the UX needs completion without polling
• [ ] Failure consumer maps sessionId / connectionId back to the IDE
• [ ] Alarm on destination delivery failures + failure-queue depth
• [ ] IAM: worker can publish to destination; least privilege ARNs
• [ ] Prefer SQS FIFO for mutating tools; destinations for remaining Event paths
• [ ] Keep async retries low; pair with recursive-loop and depth guards
• [ ] Verify with a forced failure (bad IAM) and confirm a message lands

Async without destinations is a production bug waiting for a quiet Friday. Wire OnFailure so every failed agent tool invoke becomes a message you can replay — not a shrug in CloudWatch Logs.

Related: Lambda Destinations: Async Failure Routing Beyond Plain SQS DLQs; AWS Lambda Coding Agent Sandbox: Safe Tool Isolation for LLM Agents; Lambda Warm Pools: Low-Latency Backends for Coding Agent Tools.