CloudTrail Lake: Query Agent IAM Abuses Without Spreadsheets

A coding agent with a broad task role will eventually do something you did not intend: s3:ListAllMyBuckets from a “read file” tool, kms:Decrypt on the wrong key, or iam:PassRole into a privileged sandbox. CloudTrail event history in the console is fine for one incident; it collapses when you need “every AccessDenied and unexpected allow for role/agent-tool-* across seven days and three accounts.” CloudTrail Lake is the unfair advantage — managed SQL over immutable management (and optionally data) events so you hunt IAM abuse like a query, not a CSV export. Pair with KMS Decrypt Grants and Verified Permissions for prevention — this post is the detection query layer.

⚡ TL;DR: Create an organization or account-level CloudTrail Lake event data store for management events. SQL-filter on userIdentity.sessionContext.sessionIssuer.userName / ARNs matching agent roles, errorCode = AccessDenied, and sensitive eventName allowlists. Alert via EventBridge or scheduled Athena-style reviews. Related: KMS grants, IAM condition keys for agents, Logs Insights forensics.

What CloudTrail Lake is (and is not)

CloudTrail Lake stores events in event data stores you query with SQL-like syntax (not classic Athena DDL on your own S3). You pay for ingestion + retention + queries. It is not a replacement for application logs (Logs Insights) and not Cedar authorization (Verified Permissions). Use Lake when the question is: “Which principal called which AWS API?”

Stand up an event data store for agent forensics

Prefer an organization trail feeding Lake if you run multi-account agent sandboxes (dev/staging/prod + per-tenant accounts). For a single account MVP:

# ✅ Create management-event data store (CLI sketch — adjust retention)
aws cloudtrail create-event-data-store \
  --name agent-iam-forensics \
  --retention-period 90 \
  --multi-region-enabled \
  --organization-enabled \
  --advanced-event-selectors '[
    {
      "Name": "ManagementEventsAll",
      "FieldSelectors": [
        {"Field": "eventCategory", "Equals": ["Management"]}
      ]
    }
  ]'

# ❌ Relying only on console "Event history" 90-day UI for weekly reviews
# You will miss cross-account patterns and cannot join on role name at scale

Optionally add data events for S3 object-level or Lambda invoke if your threat model includes exfiltration via GetObject on the wrong prefix — data events cost more; start with management events + tight IAM, then expand.

SQL queries that catch agent IAM abuse

CloudTrail Lake query language resembles Athena. Filter on agent role name patterns you already enforce with IAM condition keys / tags.

-- AccessDenied volume by agent role (last 7 days)
SELECT
  userIdentity.sessionContext.sessionIssuer.userName AS role_name,
  eventName,
  errorCode,
  count(*) AS n
FROM alias_agent_iam_forensics
WHERE eventTime >= DATE_ADD('day', -7, CURRENT_TIMESTAMP)
  AND userIdentity.sessionContext.sessionIssuer.userName LIKE 'agent-tool-%'
  AND errorCode = 'AccessDenied'
GROUP BY 1, 2, 3
ORDER BY n DESC
LIMIT 100;

-- Unexpected allows: sensitive APIs that should never appear for read-only tools
SELECT
  eventTime,
  userIdentity.sessionContext.sessionIssuer.userName AS role_name,
  eventName,
  sourceIPAddress,
  userAgent,
  requestParameters
FROM alias_agent_iam_forensics
WHERE eventTime >= DATE_ADD('day', -1, CURRENT_TIMESTAMP)
  AND userIdentity.sessionContext.sessionIssuer.userName LIKE 'agent-tool-%'
  AND errorCode IS NULL
  AND eventName IN (
    'DeleteBucket', 'PutBucketPolicy', 'CreateUser', 'AttachRolePolicy',
    'PutRolePolicy', 'CreateAccessKey', 'PassRole'
  )
ORDER BY eventTime DESC;

-- KMS Decrypt callers for agent roles (pair with grant audits)
SELECT
  eventTime,
  userIdentity.arn,
  eventName,
  json_extract_scalar(requestParameters, '$.keyId') AS key_id,
  errorCode
FROM alias_agent_iam_forensics
WHERE eventTime >= DATE_ADD('day', -3, CURRENT_TIMESTAMP)
  AND eventSource = 'kms.amazonaws.com'
  AND eventName IN ('Decrypt', 'GenerateDataKey')
  AND userIdentity.sessionContext.sessionIssuer.userName LIKE 'agent-%'
ORDER BY eventTime DESC
LIMIT 200;

Save these as Lake saved queries. When PassRole or AttachRolePolicy appears for an agent role, treat it as a severity — agents should not mutate IAM.

Wire alerts without a spreadsheet ritual

1. Scheduled review: EventBridge Scheduler (overnight batches pattern) invokes a Lambda that runs Lake StartQuery / GetQueryResults, posts Slack if sensitive eventName count > 0.
2. Real-time complement: CloudTrail → EventBridge rules on DeleteBucket, AttachRolePolicy for faster paging; Lake for historical join/aggregate.
3. Correlate: Join Lake requestId / time windows with ADOT traces and tool session_id from app logs when investigating a single abuse window.

// ✅ Sketch: start Lake query from Lambda (pseudo)
import {
  CloudTrailClient,
  StartQueryCommand,
  GetQueryResultsCommand,
} from "@aws-sdk/client-cloudtrail";

const ct = new CloudTrailClient({});

export async function runDeniedQuery(eventDataStoreId: string) {
  const start = await ct.send(
    new StartQueryCommand({
      QueryStatement: `
        SELECT eventName, count(*) n
        FROM ${eventDataStoreId}
        WHERE errorCode = 'AccessDenied'
          AND eventTime >= DATE_ADD('day', -1, CURRENT_TIMESTAMP)
          AND userIdentity.sessionContext.sessionIssuer.userName LIKE 'agent-tool-%'
        GROUP BY eventName ORDER BY n DESC LIMIT 20
      `,
    })
  );
  // poll GetQueryResults with start.QueryId — omit busy-wait details
  return start.QueryId;
}

Hardening loop: from query hit to policy fix

Prevention still wins: Cedar for tool authorization, IAM path/tag conditions for runtime, AppConfig kill switches when a tool goes sideways. Lake tells you when prevention failed.

Production checklist

• [ ] Event data store covers all accounts where agent roles live (org store preferred).
• [ ] Agent roles follow a naming/tag convention queryable in SQL.
• [ ] Saved queries for AccessDenied, sensitive allows, KMS Decrypt.
• [ ] Daily/hourly Lambda or Scheduler job; not “someone opens the console.”
• [ ] Runbooks map findings to IAM/Cedar/KMS grant changes.
• [ ] Data events enabled only where threat model justifies cost.
• [ ] Retention ≥ your incident response SLA (90 days is a common start).

CloudTrail Lake will not stop a bad IAM policy — it stops you from hunting agent abuse with screenshots and spreadsheets while the role is still live.

Sample investigation playbook (30 minutes)

When Slack fires on sensitive eventName or a spike in AccessDenied:

1. Scope the principal — copy userIdentity.arn and sessionIssuer.userName from the Lake row. Confirm it matches an agent tool role, not a human break-glass user.
2. Pull the window — re-query ±15 minutes around eventTime for that role; look for preceding AssumeRole and following data-plane hints.
3. Map to tool — join with app logs on approximate time + sourceIPAddress / VPC endpoint; your tool Lambda should log tool_name and session_id (Logs Insights).
4. Contain — AppConfig kill switch for that tool; optionally detach inline policy or deny with SCPs if org-level.
5. Fix forward — tighten IAM resource ARNs, add Cedar permit constraints, rotate if credentials leaked.
6. Write the finding — one paragraph in the runbook: query used, root cause, PR links.

Do not start by downloading 50k events to Excel. Start with the saved query that already groups by eventName.

Cost and retention notes

Lake pricing is ingestion + storage + queries. Management events for a busy multi-tenant agent platform are usually modest compared to Bedrock token spend — but data events on high-churn S3 prefixes can dominate. Start management-only; add S3 data events for the artifact bucket prefix only. Set retention to match compliance (90/365); shorter retention saves money but blinds postmortems. Cap automated query fan-out so a buggy Scheduler job cannot run unbounded SQL every minute.