Day 39: Observability: Traces Across Prompt, Retrieval, Tools

“The agent was weird” is not a bug report. You need a trace_id that stitches prompt build, retrieval, model latency, tool calls, and citation gate decisions into one waterfall. Day 39 defines the minimum span set and redaction rules.

⚡ TL;DR: One trace per user turn. Spans: retrieve, prompt.build, converse, tool.*, cite.gate. Attributes for model id, index sha, tokens — not secrets. 100% sample errors.

Minimum span set

from opentelemetry import trace
tracer = trace.get_tracer("ai.gateway")

def handle_turn(req):
    with tracer.start_as_current_span("agent.turn") as root:
        root.set_attribute("session.id", req.session_id)
        with tracer.start_as_current_span("retrieve") as sp:
            chunks = retrieve(req.question)
            sp.set_attribute("retrieve.k", len(chunks))
            sp.set_attribute("index.sha", INDEX_SHA)
        with tracer.start_as_current_span("converse") as sp:
            sp.set_attribute("model.id", MODEL)
            out = converse(...)
            sp.set_attribute("usage.input_tokens", out["usage"]["inputTokens"])
        with tracer.start_as_current_span("cite.gate") as sp:
            gated = gate(out, chunks)
            sp.set_attribute("cite.ok", not gated.get("refuse"))
        return gated

Store full prompts in a locked, TTL’d debug store if needed — not in world-readable logs. Attach failure.cls from Day 29 when known.

Production checklist

• [ ] W3C traceparent through API → workers
• [ ] Span attribute schema documented
• [ ] PII/secret redaction reviewed
• [ ] Trace-linked examples in Sev runbooks
• [ ] Sampling: 100% errors, partial success
• [ ] Support tool can replay retrieval ids by trace

Series navigation

← Day 38 · Day 40 →

Last updated September 11, 2026