ADOT OpenTelemetry: Trace Multi-Hop Agent Tool Calls Across Lambda

A coding agent turn is rarely one Lambda. Planner → tool runner → sandbox → summarizer. When p99 spikes, CloudWatch Logs Insights across four function names is archaeology. AWS Distro for OpenTelemetry (ADOT) on Lambda propagates W3C traceparent so one trace ID stitches the whole tool graph — and span attributes carry tool.name, tenant.id, and token counts without drowning in EMF-only metrics.

⚡ TL;DR: Instrument agent orchestrator and tool Lambdas with ADOT; pass traceparent / tracestate on every Invoke and HTTP tool call. Add span attributes for tool name, idempotency key, and tokens. Keep per-tenant cost in CloudWatch EMF; use traces for latency and error causality. Pair with Lambda Destinations for async failure routing.

Why logs alone fail for agent graphs

Agent graphs fan out. Map states invoke five tools. One tool fails after a retry. Without distributed context you cannot answer: which planner decision caused this shell_exec? Traces give you the causal tree; metrics give you the bill; logs give you the payload (carefully redacted).

// ❌ No context on nested Invoke — new trace every hop
await lambda.send(
  new InvokeCommand({
    FunctionName: "agent-tool-runner",
    Payload: Buffer.from(JSON.stringify({ tool: call })),
  })
);

✅ Inject W3C headers into the payload envelope or ClientContext, and extract them in the child.

ADOT on Lambda (Node.js)

Use the ADOT Lambda layer + OpenTelemetry SDK. Prefer the managed layer over bundling the world into your zip.

// handler instrumented with OTel API (ADOT layer provides the SDK)
import { trace, context, propagation, SpanStatusCode } from "@opentelemetry/api";

const tracer = trace.getTracer("coding-agent", "1.0.0");

type ToolCall = {
  name: string;
  args: Record<string, unknown>;
  idempotencyKey: string;
  tenantId: string;
  tokensIn?: number;
  tokensOut?: number;
};

export async function runTool(call: ToolCall) {
  return tracer.startActiveSpan(
    `tool.${call.name}`,
    {
      attributes: {
        "faas.name": process.env.AWS_LAMBDA_FUNCTION_NAME!,
        "tool.name": call.name,
        "tool.idempotency_key": call.idempotencyKey,
        "tenant.id": call.tenantId,
        "llm.tokens.in": call.tokensIn ?? 0,
        "llm.tokens.out": call.tokensOut ?? 0,
      },
    },
    async (span) => {
      try {
        const result = await executeTool(call);
        span.setAttribute("tool.ok", true);
        return result;
      } catch (err) {
        span.recordException(err as Error);
        span.setStatus({ code: SpanStatusCode.ERROR, message: String(err) });
        throw err;
      } finally {
        span.end();
      }
    }
  );
}

Export to X-Ray via ADOT (familiar console) or to a collector → Amazon Managed Prometheus / third-party. For most AWS-heavy teams, X-Ray service map on agent functions is enough to start.

Propagating traceparent across Invoke

Lambda Invoke does not automatically forward HTTP W3C headers. You must carry context in the payload (or ClientContext, which is small and base64-limited).

import { LambdaClient, InvokeCommand } from "@aws-sdk/client-lambda";
import { propagation, context as otelCtx } from "@opentelemetry/api";

const lambda = new LambdaClient({});

export async function invokeToolLambda(functionName: string, toolCall: ToolCall) {
  const carrier: Record<string, string> = {};
  propagation.inject(otelCtx.active(), carrier);

  const payload = {
    toolCall,
    _otel: {
      traceparent: carrier["traceparent"],
      tracestate: carrier["tracestate"],
    },
  };

  return lambda.send(
    new InvokeCommand({
      FunctionName: functionName,
      InvocationType: "RequestResponse",
      Payload: Buffer.from(JSON.stringify(payload)),
    })
  );
}

// Child handler extract
export function extractIncoming(event: { _otel?: { traceparent?: string; tracestate?: string } }) {
  const carrier: Record<string, string> = {};
  if (event._otel?.traceparent) carrier["traceparent"] = event._otel.traceparent;
  if (event._otel?.tracestate) carrier["tracestate"] = event._otel.tracestate;
  return propagation.extract(otelCtx.active(), carrier);
}

// ✅ Child wraps work in extracted context
export async function handler(event: any) {
  const ctx = extractIncoming(event);
  return otelCtx.with(ctx, () => runTool(event.toolCall));
}

For HTTP tool endpoints (API Gateway), rely on standard W3C headers — ADOT instrumentation often handles inbound HTTP automatically when configured.

Span attributes that matter for agents

Keep attributes low-cardinality for indexes; put high-cardinality values carefully.

❌ Putting full prompts or file contents into span events will leak secrets and blow X-Ray payload limits. Redact like you would for Bedrock ApplyGuardrail outputs.

Traces vs EMF cost metrics

CloudWatch EMF for LLM cost answers how much did tenant X spend? Traces answer why did this turn take 12s? Do both:

• EMF: TokensIn, TokensOut, ToolLatencyMs as metrics with TenantId, ToolName dimensions (bounded)
• OTel spans: full causal path, retries, Map fan-out siblings
• Destinations / DLQ: Lambda Destinations when async Invoke fails after retries

Sampling: use parent-based sampling so a sampled planner keeps tool children. Head-sample at the API Gateway entry if volume is huge; never sample only the tool runners or you lose the stitch.

Correlating Map fan-out and async Invoke

When Step Functions Map runs five tool Lambdas, each child should share the same root trace as the planner. Inject context in the Map item payload the same way as direct Invoke. For async InvocationType=Event, still inject _otel — and rely on Lambda Destinations so a failed child still emits a span before the destination fires.

// ✅ Annotate Map item with parent context from the state machine input
type MapItem = ToolCall & {
  _otel?: { traceparent?: string; tracestate?: string };
};

export function attachTraceToItems(
  items: ToolCall[],
  carrier: Record<string, string>
): MapItem[] {
  return items.map((item) => ({
    ...item,
    _otel: {
      traceparent: carrier["traceparent"],
      tracestate: carrier["tracestate"],
    },
  }));
}

X-Ray groups subsegments under one trace ID; in pure OTel backends you will see a parent span FanOutTools with five tool.* children. That is the debugging unfair advantage over grepping four log groups.

Also emit a single EMF metric line per turn for cost (tenant tokens) while keeping high-cardinality idempotency keys out of metric dimensions — traces hold them as span attributes instead.

Operational checklist

• [ ] ADOT layer on orchestrator + all tool Lambdas
• [ ] Inject/extract traceparent on every Lambda Invoke envelope
• [ ] Span names tool.{name}; attributes for tenant, tokens, idempotency key
• [ ] No raw prompts/secrets in span events
• [ ] EMF for cost; traces for causality; Destinations for async failure
• [ ] Parent-based sampling across the agent graph
• [ ] Dashboard: X-Ray service map + EMF tenant cost + error rate by tool.name

Related reading

• CloudWatch EMF for LLM Cost Per-Tenant Token Metrics That Survive Sampling
• Lambda Destinations: Route Failed Agent Tool Invokes Without Silent Drops
• Bedrock ApplyGuardrail API: Pre/Post Filters for Tool I/O in Coding Agents
• AI Coding Agent Tool Schemas: Strict JSON Contracts That Survive Retries

Last updated on September 20, 2026