Day 76: p99 of Agents: Queueing, Not Just Model Latency

Your agent’s p99 is rarely “the model was slow.” It is queue wait for a Bedrock slot, tool fan-out on a cold Lambda, or a human approval sitting in Slack. If you only chart model latency, you will “optimize” the wrong layer and still miss the SLO.

⚡ TL;DR: Split end-to-end latency into TTFB, model time, tool wait, queue wait, and human-gate wait. Cap concurrency before you buy Provisioned Throughput. Page on queue depth and approval age — not just tokens/sec.

Decompose the timeline

Instrument every hop with OpenTelemetry spans that a human can replay:

# ✅ Emit explicit wait classes — never fold into "llm_ms"
from time import monotonic

class AgentTrace:
    def __init__(self, request_id: str):
        self.request_id = request_id
        self.marks: dict[str, float] = {}

    def mark(self, name: str) -> None:
        self.marks[name] = monotonic()

    def ms(self, a: str, b: str) -> float:
        return (self.marks[b] - self.marks[a]) * 1000

❌ A single duration_ms on the API Gateway log that mixes model + tools + human gates.

Queueing beats raw model speed

Little’s Law still applies: (L = \lambda W). If arrival rate of agent jobs exceeds service rate, p99 explodes even when p50 model latency looks fine.

# ✅ Admission control before invoke
MAX_INFLIGHT = 40  # calibrated to Bedrock on-demand headroom

def admit(inflight: int) -> str:
    if inflight >= MAX_INFLIGHT:
        return "shed_or_queue"  # Day 78
    if inflight >= int(MAX_INFLIGHT * 0.8):
        return "degrade_to_draft_model"  # Day 73
    return "full"

Human gates without silent p99 bombs

Dual control is correct (Day 17/40) and also destroys latency histograms if you leave approvals in the same timer as model calls.

• Start a separate SLO for “time-to-approval.”
• Notify owners; escalate after N minutes; auto-expire unsafe tools.
• Never block the HTTP request on a human — return 202 + session id.

Failure modes that fake “model latency”

Timeouts at API Gateway often include SQS visibility timeouts and Step Functions wait states. If your dashboard’s single Latency metric is ALB target response time, you will never see human-gate wait. Export span-derived histograms to CloudWatch EMF or Prometheus histograms with wait_class label. When p99 moves, drill by class first — buying a bigger model is the last lever, not the first.

Closing checklist

• [ ] Spans for queue / TTFT / decode / tool / human separately
• [ ] Dashboard panels per span class at p50/p95/p99
• [ ] Admission control tied to provider concurrency
• [ ] Human-gate age alert distinct from model latency alert
• [ ] Load test with realistic tool latency, not mock 5ms stubs

Series navigation

Day 75: Autoscaling Retrieval and Embed Jobs · Day 77: Caching RAG Answers Safely

Last updated September 11, 2026