23 views
This guide focuses on AI agent latency for production systems, with practical trade-offs for reliability, security, and cost.
Your agent’s p99 is rarely “the model was slow.” It is queue wait for a Bedrock slot, tool fan-out on a cold Lambda, or a human approval sitting in Slack. If you only chart model latency, you will “optimize” the wrong layer and still miss the SLO.
⚡ TL;DR: Split end-to-end latency into TTFB, model time, tool wait, queue wait, and human-gate wait. Cap concurrency before you buy Provisioned Throughput. Page on queue depth and approval age — not just tokens/sec.
AI agent latency: production guidance
Instrument every hop with OpenTelemetry spans that a human can replay:| Span | What it measures | Typical killer |
|---|---|---|
queue.wait |
Time in SQS/Step Functions before worker | Burst of PRs |
model.ttft |
Time to first token | Cold start / throttle |
model.decode |
Stream duration | Long answers |
tool.wait |
External API / sandbox | ECS pull, GitHub |
human.gate |
Dual-control approval | Off-hours on-call |
python
❌ A single # ✅ Emit explicit wait classes — never fold into "llm_ms"
from time import monotonic
class AgentTrace:
def __init__(self, request_id: str):
self.request_id = request_id
self.marks: dict[str, float] = {}
def mark(self, name: str) -> None:
self.marks[name] = monotonic()
def ms(self, a: str, b: str) -> float:
return (self.marks[b] - self.marks[a]) * 1000
duration_ms on the API Gateway log that mixes model + tools + human gates.
Queueing beats raw model speed
Little’s Law still applies: (L = \lambda W). If arrival rate of agent jobs exceeds service rate, p99 explodes even when p50 model latency looks fine.python
# ✅ Admission control before invoke
MAX_INFLIGHT = 40 # calibrated to Bedrock on-demand headroom
def admit(inflight: int) -> str:
if inflight >= MAX_INFLIGHT:
return "shed_or_queue" # Day 78
if inflight >= int(MAX_INFLIGHT * 0.8):
return "degrade_to_draft_model" # Day 73
return "full"
Human gates without silent p99 bombs
Dual control is correct (Day 17/40) and also destroys latency histograms if you leave approvals in the same timer as model calls.- Start a separate SLO for “time-to-approval.”
- Notify owners; escalate after N minutes; auto-expire unsafe tools.
- Never block the HTTP request on a human — return
202+ session id.
Failure modes that fake “model latency”
Timeouts at API Gateway often include SQS visibility timeouts and Step Functions wait states. If your dashboard’s singleLatency metric is ALB target response time, you will never see human-gate wait. Export span-derived histograms to CloudWatch EMF or Prometheus histograms with wait_class label. When p99 moves, drill by class first — buying a bigger model is the last lever, not the first.
Closing checklist
- [ ] Spans for queue / TTFT / decode / tool / human separately
- [ ] Dashboard panels per span class at p50/p95/p99
- [ ] Admission control tied to provider concurrency
- [ ] Human-gate age alert distinct from model latency alert
- [ ] Load test with realistic tool latency, not mock 5ms stubs
Series navigation
Day 75: Autoscaling Retrieval and Embed Jobs · Day 77: Caching RAG Answers Safely Last updated September 11, 2026 Related: IAM for AI Agents: Condition Keys That Limit Blast Radius by Tag; Day 82: Voice and Realtime Agents; Day 51: Multi-Agent Roles: Planner, Implementer, Critic.Watch: 100 Days of AI on YouTube
Short videos from CheatCoders, one AI engineering topic per day.
Most viewed
- Python Decorators Explained: From Simple Wrappers to Production Patterns
- Claude Code vs Cursor vs GitHub Copilot: Choose by Workflow, and Set Up One Repo That Works With All Three
- AI Agent Frameworks in 2025: LangGraph vs CrewAI vs AutoGen vs Raw API
- Python String Methods: Every str Method With Real Production Examples
- Css Animation Performance: Performance-Optimized Motion for Modern UIs
Newly added
- AWS Systems Manager Parameter Store: Hierarchical Runtime Config for Coding Agents
- Amazon Cognito Identity Pools: Federate Coding-Agent Tool Callers Without Long-Lived Keys
- Amazon Athena for Coding Agents: Query Tool Audit Logs and Eval Traces on S3 Without a Warehouse
- AWS Batch for Coding Agents: Parallel Overnight PR Reviews and Long-Running Jobs Without Lambda Timeouts
- Amazon Bedrock Prompt Management: Version and A/B Test Coding-Agent System Prompts
Deep-dive PDF
Get the expanded guide for this post — extra diagrams-style checklists, failure modes, and a production walkthrough. Free when you subscribe to CheatCoders.
Already subscribed? or open the subscribe page.
Discover more from CheatCoders
Subscribe to get the latest posts sent to your email.
