Lambda Recursive Loop Protection: Stop Agent Self-Invokes From Burning Quotas

0 views

Your coding agent has a tool that calls lambda:InvokeFunction. That tool exists so the agent can fan out a lint job, a sandbox build, or a second “worker” Lambda. The first time the model hallucinates “I should re-invoke myself to continue,” you discover what recursive Lambda traffic looks like on a Friday afternoon: concurrency climbs, account quotas melt, and CloudWatch fills with the same payload forever. The unfair advantage is not hoping the model behaves. It is recursive loop detection, hard depth limits on tool schemas, and circuit breakers that fail closed before quotas do.

⚡ TL;DR: AWS detects same-account Lambda→Lambda recursive loops and throttles them after ~16 invocations in a chain. Agent tool graphs trip this constantly — especially when a function invokes its own alias. Pair detection with maxDepth in AI Coding Agent Tool Schemas, DLQs + alarms from SQS FIFO + Lambda Ordered Agent Job Queues, and blast-radius tags in IAM Condition Keys for Agent Runtimes. Never let an agent invoke its own function name or alias without an explicit depth budget.

How AWS recursive loop detection works

For supported runtimes and invoke paths, Lambda tracks a recursion context across same-account Invoke calls. When the chain of Lambda→Lambda invocations exceeds the detection threshold (documented around 16), further invokes in that loop are throttled and you get RecursiveInvocationException-class failures plus CloudWatch metrics under Recursive Invocations Dropped.

Key constraints agents miss:

  • Detection is same-account, Lambda-to-Lambda. Cross-account fan-out or Lambda→SQS→Lambda may not trip the same guard the way you expect.
  • An alias or version of the same function still counts as the function. Agents that “call the prod alias of myself” are the classic self-own.
  • Detection is a safety net, not an architecture. By the time it fires you have already spent concurrency and money.
// ❌ Agent tool that can invoke the same function forever
import { LambdaClient, InvokeCommand } from "@aws-sdk/client-lambda";

const lambda = new LambdaClient({});

export async function invokeWorkerTool(input: {
  functionName: string;
  payload: unknown;
}) {
  // No depth, no caller identity, no circuit breaker
  await lambda.send(
    new InvokeCommand({
      FunctionName: input.functionName, // model can pass THIS function's ARN
      InvocationType: "Event",
      Payload: Buffer.from(JSON.stringify(input.payload)),
    })
  );
  return { ok: true };
}

✅ Treat every agent-driven InvokeFunction as untrusted input: allowlist ARNs, require depth, refuse self.

How agent tool graphs trip the loop

Coding agents do not need malice. Common tripwires:

  1. Self-continue: “Invoke this Lambda again with the next file” — the tool name is run_worker, the ARN is this function.
  2. Alias bounce: $LATEST invokes prod alias of the same function; detection still sees a loop.
  3. Fan-out without join: Parent invokes N children; a buggy child re-invokes the parent “for status.”
  4. Retry + async invoke: At-least-once + recursive Event invoke doubles the graph under throttle.

Map this to your tool schema. If InvokeFunction is a tool, the schema must carry depth and parentInvocationId the same way you carry idempotency keys for Bedrock tool results.

// ✅ Tool schema with hard depth + allowlisted targets
export const invokeWorkerToolSchema = {
  name: "invoke_worker",
  description: "Invoke an allowlisted worker Lambda. Refuses self and depth>max.",
  inputSchema: {
    type: "object",
    additionalProperties: false,
    required: ["targetKey", "payload", "depth", "idempotencyKey"],
    properties: {
      targetKey: {
        type: "string",
        enum: ["lint", "test", "index"], // never raw ARN from the model
      },
      payload: { type: "object" },
      depth: { type: "integer", minimum: 0, maximum: 3 },
      idempotencyKey: { type: "string", minLength: 8, maxLength: 128 },
      parentInvocationId: { type: "string" },
    },
  },
} as const;

const TARGETS: Record<string, string> = {
  lint: process.env.LINT_FN_ARN!,
  test: process.env.TEST_FN_ARN!,
  index: process.env.INDEX_FN_ARN!,
};

const SELF_ARNS = new Set(
  [process.env.AWS_LAMBDA_FUNCTION_NAME, process.env.SELF_FN_ARN].filter(Boolean)
);

export async function invokeWorkerTool(input: {
  targetKey: keyof typeof TARGETS;
  payload: unknown;
  depth: number;
  idempotencyKey: string;
  parentInvocationId?: string;
}) {
  if (input.depth > 3) {
    return { ok: false, error: "maxDepth_exceeded" };
  }
  const arn = TARGETS[input.targetKey];
  if (!arn) return { ok: false, error: "unknown_target" };

  // ❌ Never allow model-supplied FunctionName that equals self
  for (const self of SELF_ARNS) {
    if (arn.includes(String(self))) {
      return { ok: false, error: "refused_self_invoke" };
    }
  }

  await lambda.send(
    new InvokeCommand({
      FunctionName: arn,
      InvocationType: "Event",
      Payload: Buffer.from(
        JSON.stringify({
          ...input.payload,
          depth: input.depth + 1,
          idempotencyKey: input.idempotencyKey,
          parentInvocationId: input.parentInvocationId,
        })
      ),
    })
  );
  return { ok: true, targetKey: input.targetKey, nextDepth: input.depth + 1 };
}

Circuit breakers in the handler

Depth in the schema is necessary but not sufficient. Put a breaker in the hot path:

import { DynamoDBClient } from "@aws-sdk/client-dynamodb";
import {
  DynamoDBDocumentClient,
  UpdateCommand,
} from "@aws-sdk/lib-dynamodb";

const ddb = DynamoDBDocumentClient.from(new DynamoDBClient({}));
const TABLE = process.env.CIRCUIT_TABLE!;
const MAX_PER_SESSION = 20;

export async function checkCircuit(sessionId: string): Promise<boolean> {
  const out = await ddb.send(
    new UpdateCommand({
      TableName: TABLE,
      Key: { pk: `session#${sessionId}`, sk: "invoke_budget" },
      UpdateExpression:
        "ADD #c :one SET #ttl = if_not_exists(#ttl, :ttl)",
      ConditionExpression: "attribute_not_exists(#c) OR #c < :max",
      ExpressionAttributeNames: { "#c": "count", "#ttl": "ttl" },
      ExpressionAttributeValues: {
        ":one": 1,
        ":max": MAX_PER_SESSION,
        ":ttl": Math.floor(Date.now() / 1000) + 3600,
      },
      ReturnValues: "UPDATED_NEW",
    })
  );
  return (out.Attributes?.count as number) <= MAX_PER_SESSION;
}

✅ Fail closed on ConditionalCheckFailedException — return a tool error the model can see, do not retry-invoke.

Wire the same session budget into EMF metrics so you can alarm before recursive detection drops traffic. See CloudWatch EMF for LLM Cost for the metering pattern; add a dimension RecursiveRisk=true when depth ≥ 2.

DLQ + CloudWatch alarms that actually fire

Async invokes that get dropped or fail after retries should land in a DLQ. For agent workers:

  1. Configure on-failure destination (SQS or SNS) on the function, not only “hope Event fails quietly.”
  2. Alarm on Recursive Invocations Dropped (AWS/Lambda) and on your custom AgentInvokeRefused EMF count.
  3. Alarm on DLQ ApproximateNumberOfMessagesVisible > 0 for the worker queue.
// CDK sketch — destinations + recursive-aware alarm
import * as lambda from "aws-cdk-lib/aws-lambda";
import * as cloudwatch from "aws-cdk-lib/aws-cloudwatch";
import * as sqs from "aws-cdk-lib/aws-sqs";

const dlq = new sqs.Queue(this, "AgentWorkerDlq");

const worker = new lambda.Function(this, "AgentWorker", {
  // ...
});
worker.configureAsyncInvoke({
  maxEventAge: cdk.Duration.hours(1),
  retryAttempts: 1, // ✅ keep retries low for recursive graphs
  onFailure: new lambda.SqsDestination(dlq),
});

new cloudwatch.Alarm(this, "RecursiveDropped", {
  metric: worker.metric("RecursiveInvocationsDropped", {
    statistic: "Sum",
    period: cdk.Duration.minutes(1),
  }),
  threshold: 1,
  evaluationPeriods: 1,
  alarmDescription: "Lambda recursive loop protection dropped invokes",
});

❌ Patterns that burn quotas

Pattern Why it hurts Fix
Agent passes raw FunctionName Model can name itself Allowlist targetKey → ARN map
Invoke own alias “for continuation” Classic recursive loop Refuse self; use Step Functions / queue
InvocationType: Event + unlimited depth Silent fan-out storm maxDepth + session circuit
Cross-tool recursion (A→B→A) Detection may lag / miss intent Shared session budget in DynamoDB
Ignoring RecursiveInvocationsDropped You learn from the bill Alarm at 1

Prefer SQS + Lambda or Step Functions for multi-step agent work instead of Lambda→Lambda Event invokes. Ordered agent jobs belong on FIFO queues with idempotent handlers — not recursive self-invokes.

Checklist

  • [ ] Tool schema includes depth (max ≤ 3) and idempotencyKey
  • [ ] FunctionName is never model-supplied raw ARN; use allowlisted keys
  • [ ] Handler refuses invokes to self name / self ARN / own aliases
  • [ ] Session-level circuit breaker in DynamoDB (fail closed)
  • [ ] Async on-failure destination → DLQ + alarm
  • [ ] CloudWatch alarm on RecursiveInvocationsDropped ≥ 1
  • [ ] Prefer SQS / Step Functions over Lambda→Lambda for agent fan-out
  • [ ] IAM: agent role cannot lambda:InvokeFunction on * — tag-conditioned ARNs only

Recursive loop protection is the last line of defense. Build as if it did not exist, then keep it on so a single hallucinated self-invoke cannot torch your concurrency quota.


Discover more from CheatCoders

Subscribe to get the latest posts sent to your email.

Comments

No comments yet. Why don’t you start the discussion?

Leave a comment

No account needed. Name and email are optional.