OpenTelemetry for LLMs: Trace Prompt Latency Across Microservices

When an IDE agent feels “slow,” the bottleneck might be retrieval, a tool Lambda cold start, Bedrock queueing, or your own JSON schema validator. Without distributed traces, teams guess and scale the wrong tier. OpenTelemetry for LLM workflows means W3C trace context across every hop — proxy, tools, Bedrock, and databases — with attributes that make token latency debuggable.

⚡ TL;DR: Start a root span per user action (agent.turn). Propagate traceparent through HTTP, SQS, and Bedrock request metadata. Record gen_ai.* attributes (model, input/output tokens, cache hit). Tail-sample errors and slow turns. Illustrative debug win: cut MTTR for “assistant p99 > 8s” from hours to minutes by seeing tool vs model breakdown.

Span model for an agent turn

agent.turn (root)
  ├─ context.assemble
  ├─ retrieve.kb
  ├─ gen_ai.chat (Bedrock Converse)
  │    └─ gen_ai.tool.search_repo
  │         └─ aws.lambda.invoke
  └─ patch.validate

import { trace, SpanStatusCode, context, propagation } from "@opentelemetry/api";

const tracer = trace.getTracer("cheatcoders-agent");

export async function runTurn(userId: string, prompt: string) {
  return tracer.startActiveSpan("agent.turn", async (span) => {
    span.setAttribute("user.id", userId);
    span.setAttribute("agent.prompt_chars", prompt.length);
    try {
      const answer = await converseWithTools(prompt);
      span.setStatus({ code: SpanStatusCode.OK });
      return answer;
    } catch (err) {
      span.recordException(err as Error);
      span.setStatus({ code: SpanStatusCode.ERROR });
      throw err;
    } finally {
      span.end();
    }
  });
}

Instrument Bedrock with gen_ai attributes

import {
  BedrockRuntimeClient,
  ConverseCommand,
} from "@aws-sdk/client-bedrock-runtime";

export async function converse(modelId: string, messages: any[]) {
  return tracer.startActiveSpan("gen_ai.chat", async (span) => {
    span.setAttribute("gen_ai.system", "aws.bedrock");
    span.setAttribute("gen_ai.request.model", modelId);
    const started = Date.now();
    const resp = await new BedrockRuntimeClient({}).send(
      new ConverseCommand({
        modelId,
        messages,
        inferenceConfig: { maxTokens: 2048 },
      })
    );
    span.setAttribute("gen_ai.response.model", modelId);
    span.setAttribute("gen_ai.usage.input_tokens", resp.usage?.inputTokens ?? 0);
    span.setAttribute("gen_ai.usage.output_tokens", resp.usage?.outputTokens ?? 0);
    span.setAttribute("gen_ai.latency_ms", Date.now() - started);
    span.end();
    return resp;
  });
}

Propagate context into tool Lambdas via HTTP headers or the Lambda event envelope. For X-Ray coexistence, use ADOT layers carefully — same tradeoffs discussed in Lambda tracing posts on CheatCoders.

Propagate across queues and AppSync

// Inject W3C headers when enqueueing tool work
const carrier: Record<string, string> = {};
propagation.inject(context.active(), carrier);
await sqs.send({
  QueueUrl,
  MessageBody: JSON.stringify({ args }),
  MessageAttributes: {
    traceparent: { DataType: "String", StringValue: carrier["traceparent"] },
  },
});

On the consumer, propagation.extract before starting the child span. Missing this link is why service maps “lie” during incidents.

Tail sampling and cost

At IDE volume you cannot keep 100% of successful fast turns.

# illustrative collector policy
policies:
  - name: errors
    type: status_code
    status_code: ERROR
  - name: slow-agent
    type: latency
    threshold_ms: 5000
  - name: baseline
    type: probabilistic
    sampling_percentage: 5

Export to Amazon Managed Prometheus / Grafana or X-Ray via ADOT. Alert on gen_ai.chat p95 and tool error rates — not just host CPU. Pair with LLM cost controls so latency and spend dashboards sit side by side.

Correlate IDE, CI, and cloud backends

Developers feel latency in the IDE; the spans often live in CI runners and AWS. Carry a session.id / pr.number baggage key from the first hop:

import { propagation, context } from "@opentelemetry/api";

export function withPrBaggage<T>(pr: number, fn: () => Promise<T>) {
  // Use baggage API in your SDK version; illustrative attribute mirror:
  return tracer.startActiveSpan("ci.agent.job", async (span) => {
    span.setAttribute("ci.pr", pr);
    span.setAttribute("ci.pipeline", process.env.GITHUB_WORKFLOW ?? "local");
    try {
      return await fn();
    } finally {
      span.end();
    }
  });
}

When p99 spikes, filter traces by gen_ai.request.model and tool.name before blaming “Bedrock is slow.” Cold tool Lambdas show up as child gaps — fix with provisioned concurrency schedules, not bigger models. Pair with Lambda cold-start discipline when tools run on Lambda.

Closing checklist

✅ Dos
– ✅ One root span per agent turn with stable name
– ✅ Record model, tokens, cache hit, tool name attributes
– ✅ Propagate traceparent across HTTP, SQS, Lambda
– ✅ Tail-sample errors and slow turns
– ✅ Dashboard tool vs model latency breakdown

❌ Don’ts
– ❌ Don’t log raw prompts containing secrets into span attributes
– ❌ Don’t rely on logs alone for cross-service LLM latency
– ❌ Don’t sample only at the edge and lose tool children
– ❌ Don’t attach megabytes of retrieval chunks onto spans
– ❌ Don’t forget AppSync/API Gateway header allowlists for trace context

Related reading

• Amazon Bedrock Agents: Tool Use, Memory, and Production Guardrails
• LLM Coding Agents on AWS: Safe Tool Sandboxes with Lambda
• LLM Cost Controls: Token Budgets Per PR and Per Engineer (companion)
• Node.js Event Loop Lag: Catch p99 Stalls Before Users Feel Them

Last updated on September 11, 2026