CloudWatch Logs Insights: Multi-Tenant Agent Tool Failure Forensics

When ten tenants share one agent runtime, a spike in tool_timeout is not a single bug — it is a distribution. Raw CloudWatch Logs show interleaved JSON from Lambda; without structured fields and Insights queries you blame Bedrock, then Redis, then “the model,” while the real offender is tenant acme saturating run_tests for 45 seconds. CloudWatch Logs Insights is the unfair advantage: parse once, filter by tenant and tool, stats latency percentiles, save the query, and optionally layer Contributor Insights for top-N offenders. Pair it with ADOT OpenTelemetry for cross-hop traces and CloudWatch EMF for LLM cost for spend — this post is the failure forensics layer.

⚡ TL;DR: Emit structured JSON with tenant_id, tool_name, latency_ms, error_class from every tool Lambda. Use Logs Insights filter/stats/parse (and Saved queries) to answer “which tenant × tool is failing right now?” before you open X-Ray. Optional Contributor Insights for top offenders. Related: ADOT multi-hop traces, EMF per-tenant tokens, AppConfig kill switches.

Instrument tool Lambdas for queryable fields

Insights is only as good as your log shape. Prefer one JSON object per tool invocation (success or failure), not free-text stack dumps alone.

// ✅ Structured tool outcome — every field Insights will filter/stats on
type ToolLog = {
  msg: "tool_outcome";
  tenant_id: string;
  session_id: string;
  tool_name: string;
  latency_ms: number;
  ok: boolean;
  error_class?: "timeout" | "validation" | "upstream_5xx" | "auth" | "unknown";
  http_status?: number;
  tokens_in?: number;
  tokens_out?: number;
};

export function logToolOutcome(row: ToolLog) {
  console.log(JSON.stringify(row));
}

// ❌ Unstructured — un-queryable under load
console.log(`tool ${name} failed for ${tenant}: ${err.message}`);

Keep field names stable across planner, tool-runner, and gateway Lambdas. If you already emit EMF for cost, keep failure forensics in the same log group or a sibling .../tools group — do not invent a third taxonomy mid-incident.

Core Insights queries for agent forensics

Assume log group /aws/lambda/agent-tool-runner. Start with failure rate by tenant × tool, then latency, then error class.

# Failure rate by tenant × tool (last 1h) — paste into Logs Insights
fields @timestamp, tenant_id, tool_name, ok, error_class, latency_ms
| filter msg = "tool_outcome"
| stats count() as n,
        sum(ok = 0) as failures,
        pct(latency_ms, 95) as p95_ms
  by tenant_id, tool_name
| sort failures desc
| limit 50

# Error class breakdown for one hot tenant
fields @timestamp, tool_name, error_class, latency_ms, @message
| filter msg = "tool_outcome" and tenant_id = "acme" and ok = 0
| stats count() as n by tool_name, error_class
| sort n desc

# Slow successes that look like "hangs" to users
fields @timestamp, tenant_id, tool_name, latency_ms
| filter msg = "tool_outcome" and ok = 1 and latency_ms > 8000
| sort latency_ms desc
| limit 100

When failures cluster on one tool, flip the kill switch from AppConfig feature flags instead of redeploying while you dig.

Saved queries and dashboards

Incidents are not the time to reinvent filter msg = "tool_outcome". Save three queries in the console (or IaC via CloudFormation AWS::Logs::QueryDefinition):

1. Hot failures — tenant × tool failure counts (query above).
2. Auth / validation — error_class in ["auth","validation"] (schema drift vs IAM).
3. P95 by tool — latency without success filter (catch degraded success).

Wire a CloudWatch dashboard with Logs Insights widgets pointing at those Saved queries. Add an alarm on a metric filter if you need paging:

# Metric filter pattern (JSON) — count tool failures
{ $.msg = "tool_outcome" && $.ok = false }

Publish AgentToolFailures to a custom metric with dimensions ToolName, TenantId if you need per-tenant alarms — but start with Insights; metric cardinality explodes if every tenant becomes a dimension without care.

Optional: Contributor Insights for top offenders

Contributor Insights rules can continuously rank top tenant_id or tool_name contributors from the log group without you running queries every five minutes.

{
  "Schema": {
    "Name": "CloudWatchLogRule",
    "Version": 1
  },
  "LogFormat": "JSON",
  "Contribution": {
    "Keys": ["$.tenant_id", "$.tool_name"],
    "Filters": [
      { "Match": "$.msg", "In": ["tool_outcome"] },
      { "Match": "$.ok", "In": [false, "false", 0] }
    ],
    "ValueOf": "$.latency_ms"
  },
  "AggregateOn": "Sum"
}

Use Contributor Insights for ongoing top-N; use ad-hoc Insights for deep forensics (error class, message samples, correlation with session_id). Do not replace EMF cost metrics or ADOT traces — each answers a different question.

Correlate with traces and EMF (without boiling the ocean)

Logs Insights answers “what failed and for whom.” Traces answer “where in the hop chain.” EMF answers “what did it cost.”

• Put session_id and (if present) trace_id in the same JSON row so you can jump from Insights → X-Ray/ADOT.
• When EMF shows a tenant burning tokens with low tool success, run the failure-rate query for that tenant_id — see CloudWatch EMF for LLM cost.
• Multi-hop tool chains need ADOT OpenTelemetry so latency_ms on the leaf tool is not confused with orchestrator wait time.

# Python tool runner — same schema as TS
import json, time

def run_tool(tenant_id: str, tool_name: str, fn):
    t0 = time.perf_counter()
    ok, err_class = True, None
    try:
        return fn()
    except TimeoutError:
        ok, err_class = False, "timeout"
        raise
    except Exception:
        ok, err_class = False, "unknown"
        raise
    finally:
        print(json.dumps({
            "msg": "tool_outcome",
            "tenant_id": tenant_id,
            "tool_name": tool_name,
            "latency_ms": int((time.perf_counter() - t0) * 1000),
            "ok": ok,
            "error_class": err_class,
        }))

Checklist: ship forensics this week

• [ ] Standardize tool_outcome JSON fields across all agent Lambdas
• [ ] Create three Saved Insights queries (failures, error class, p95)
• [ ] Dashboard widget + optional metric filter for paging
• [ ] Document runbook: Insights → kill switch → ADOT → EMF
• [ ] Optionally enable Contributor Insights for top tenant×tool failures
• [ ] Cap cardinality: never dimension-alarm on unbounded free-text errors

Logs Insights will not fix bad tool schemas or missing timeouts — it makes the blast radius visible. Once you can name the tenant and tool in under a minute, the rest of the stack (kill switches, Batch for long jobs, stricter tool schemas) becomes deliberate instead of reactive.