Hunting Coding-Agent IAM Abuse in CloudTrail: Athena and CloudWatch, Plus CloudTrail Lake for Existing Customers

Status update: AWS CloudTrail Lake is no longer open to new customers as of May 31, 2026. AWS says existing customers can keep using it as normal, but Lake will only get critical bug fixes and security updates, and AWS recommends migrating Lake data to Amazon CloudWatch. CloudTrail itself (trails, Insights and aggregated events) is not affected. This guide was rewritten for that situation: the main path uses a CloudTrail trail with Amazon Athena and CloudWatch, which anyone can set up, and a short section covers Lake for teams that already have it.

A coding agent with an AWS role will eventually call something you didn’t intend: listing every bucket from a “read file” tool, decrypting with the wrong KMS key, or attaching a policy to its own role. CloudTrail records each of those calls with the principal, the API and whether it succeeded. The work is turning those records into a few questions you ask every day, and alerts for the handful of calls that should never happen.

⚡ TL;DR

• Give agent roles a naming convention (for example agent-tool-*) so every query can filter on userIdentity.sessionContext.sessionIssuer.userName.
• Send an organization trail to S3 and query it with Athena, using partition projection so each query scans only the days you ask for.
• Three daily queries: denied calls by agent role, successful IAM writes by agent roles, and roles passed to new resources.
• iam:PassRole is a permission, not an API: it never appears as an event, so look at the call that used the role.
• For real-time alerts, use an EventBridge rule on specific IAM write calls and a CloudWatch Logs metric filter on denied calls.

Who this is for: platform and security engineers running coding agents with AWS credentials. The Athena queries were tested by translating them with sqlglot and running them on DuckDB over sample records in the CloudTrail format. They were not run on Athena itself.

Pick the tool by the question

Step 1: Make agent activity easy to find

When a tool assumes a role, CloudTrail records userIdentity.type as AssumedRole, and userIdentity.sessionContext.sessionIssuer holds the role’s ARN and name. If every agent role shares a prefix, every query and alert below can filter on that name. Use a role session name that includes the agent session or tool ID too: it appears at the end of userIdentity.arn (assumed-role/<role>/<session-name>), which is how you get from an event back to a specific agent run.

Create an organization trail if agents run in several accounts, so one bucket holds every account’s events. Management events cover IAM, KMS key usage and most control-plane calls. Data events, such as S3 object reads, cost extra; add them only for buckets your threat model needs.

Step 2: Query the trail with Athena

The Athena documentation provides a CREATE TABLE statement for CloudTrail logs with partition projection: the table is partitioned by a timestamp column in yyyy/MM/dd form that matches the trail’s S3 layout, so Athena computes the partitions instead of you loading them. Copy that statement from the docs (linked below) and set the bucket, account and Region. With it in place, filtering on "timestamp" limits how much data each query scans.

-- 1. Denied calls by agent role, last 7 days. The partition filter keeps the scan to 7 days of files.
SELECT useridentity.sessioncontext.sessionissuer.username AS role_name,
       eventsource, eventname, errorcode, count(*) AS n
FROM cloudtrail_logs_pp
WHERE "timestamp" >= date_format(current_date - interval '7' day, '%Y/%m/%d')
  AND useridentity.sessioncontext.sessionissuer.username LIKE 'agent-tool-%'
  AND (errorcode LIKE '%AccessDenied%' OR errorcode LIKE '%UnauthorizedOperation%')
GROUP BY 1, 2, 3, 4
ORDER BY n DESC;

-- 2. Successful IAM writes by agent roles. Any row here deserves a look.
SELECT eventtime, useridentity.sessioncontext.sessionissuer.username AS role_name,
       eventname, requestparameters
FROM cloudtrail_logs_pp
WHERE "timestamp" >= date_format(current_date - interval '1' day, '%Y/%m/%d')
  AND useridentity.sessioncontext.sessionissuer.username LIKE 'agent-tool-%'
  AND eventsource = 'iam.amazonaws.com'
  AND readonly = 'false'
  AND errorcode IS NULL
ORDER BY eventtime DESC;

-- 3. Roles handed to new resources. PassRole has no event of its own, so read the call that used it.
SELECT eventtime, useridentity.sessioncontext.sessionissuer.username AS role_name, eventsource, eventname,
       regexp_extract(requestparameters, 'arn:aws:iam::[0-9]{12}:role/[A-Za-z0-9+=,.@_/-]+') AS passed_role
FROM cloudtrail_logs_pp
WHERE "timestamp" >= date_format(current_date - interval '7' day, '%Y/%m/%d')
  AND useridentity.sessioncontext.sessionissuer.username LIKE 'agent-tool-%'
  AND errorcode IS NULL
  AND regexp_like(requestparameters, 'arn:aws:iam::[0-9]{12}:role/')
ORDER BY eventtime DESC;

The three queries answer different questions:

1. Denied calls by role. A burst of denials usually means a tool’s arguments are too open (for example, a path parameter that lets it ask for any bucket), or that someone is probing. Different services use different error codes: S3 and IAM return AccessDenied, EC2 returns Client.UnauthorizedOperation, so match both.
2. Successful IAM writes. Agent tool roles normally have no reason to change IAM. Any row is worth a look.
3. Roles passed to new resources. The IAM documentation is explicit: PassRole is a permission, not an API call, so no CloudTrail event is generated for it. To see which role was handed to a service, you read the event that created or changed the resource (here Lambda’s CreateFunction20150331) and pull the role ARN out of its request parameters.

Testing the queries without an AWS account

Athena runs Trino SQL. To test the logic, this script loads sample records in the CloudTrail shape (an assumed-role identity, as in the CloudTrail userIdentity reference) into DuckDB, translates each query from Trino to DuckDB with sqlglot, and runs it:

"""Synthetic CloudTrail records in the documented shape (userIdentity for an assumed role)."""
import json

def ev(t, role, source, name, error=None, params=None, readonly="false"):
    return {"eventVersion": "1.10", "eventTime": t, "eventSource": source, "eventName": name,
            "awsRegion": "us-east-1", "sourceIPAddress": "10.0.3.17", "userAgent": "Boto3/1.40",
            "errorCode": error, "errorMessage": "denied" if error else None,
            "requestParameters": json.dumps(params) if params else None, "readOnly": readonly,
            "eventCategory": "Management", "recipientAccountId": "111122223333",
            "userIdentity": {"type": "AssumedRole", "arn": f"arn:aws:sts::111122223333:assumed-role/{role}/sess-1",
                             "accountId": "111122223333",
                             "sessionContext": {"sessionIssuer": {"type": "Role", "arn": f"arn:aws:iam::111122223333:role/{role}",
                                                                  "accountId": "111122223333", "userName": role}}}}

records = [
    ev("2026-10-09T08:00:01Z", "agent-tool-reader", "s3.amazonaws.com", "ListBuckets", "AccessDenied", readonly="true"),
    ev("2026-10-09T08:00:02Z", "agent-tool-reader", "s3.amazonaws.com", "ListBuckets", "AccessDenied", readonly="true"),
    ev("2026-10-09T08:01:00Z", "agent-tool-reader", "ec2.amazonaws.com", "DescribeInstances", "Client.UnauthorizedOperation", readonly="true"),
    ev("2026-10-09T08:02:00Z", "agent-tool-reader", "kms.amazonaws.com", "Decrypt", readonly="true", params={"keyId": "arn:aws:kms:us-east-1:111122223333:key/prod-db"}),
    ev("2026-10-09T08:03:00Z", "agent-tool-deployer", "lambda.amazonaws.com", "CreateFunction20150331",
       params={"functionName": "agent-scratch", "role": "arn:aws:iam::111122223333:role/AdminRole"}),
    ev("2026-10-09T08:04:00Z", "agent-tool-deployer", "iam.amazonaws.com", "AttachRolePolicy",
       params={"roleName": "agent-tool-deployer", "policyArn": "arn:aws:iam::aws:policy/AdministratorAccess"}),
    ev("2026-10-09T08:05:00Z", "ci-deploy", "iam.amazonaws.com", "AttachRolePolicy",
       params={"roleName": "app", "policyArn": "arn:aws:iam::111122223333:policy/app-read"}),
]
json.dump({"Records": records}, open("sample.json", "w"))
print(len(records), "records")

"""Run the Athena (Trino) queries on DuckDB over sample records, via sqlglot, to test the logic."""
import json
import duckdb
import sqlglot

recs = json.load(open("sample.json"))["Records"]
con = duckdb.connect()
con.execute("""CREATE TABLE cloudtrail_logs_pp (eventtime VARCHAR, eventsource VARCHAR, eventname VARCHAR,
  errorcode VARCHAR, requestparameters VARCHAR, readonly VARCHAR, "timestamp" VARCHAR,
  useridentity STRUCT(arn VARCHAR, sessioncontext STRUCT(sessionissuer STRUCT(arn VARCHAR, username VARCHAR))))""")
today = con.execute("SELECT strftime(current_date, '%Y/%m/%d')").fetchone()[0]
for r in recs:
    si = r["userIdentity"]["sessionContext"]["sessionIssuer"]
    con.execute("INSERT INTO cloudtrail_logs_pp VALUES (?,?,?,?,?,?,?, {'arn': ?, 'sessioncontext': {'sessionissuer': {'arn': ?, 'username': ?}}})",
                [r["eventTime"], r["eventSource"], r["eventName"], r["errorCode"], r["requestParameters"],
                 r["readOnly"], today, r["userIdentity"]["arn"], si["arn"], si["userName"]])

for i, sql in enumerate(s for s in open("queries.sql").read().split(";") if s.strip().replace("\n", "")):
    sqlglot.parse_one(sql, read="athena")                          # must parse as Athena SQL
    duck = sqlglot.transpile(sql, read="trino", write="duckdb")[0]
    print(f"query {i + 1}:")
    for row in con.execute(duck).fetchall():
        print("  ", row)

query 1:
   ('agent-tool-reader', 's3.amazonaws.com', 'ListBuckets', 'AccessDenied', 2)
   ('agent-tool-reader', 'ec2.amazonaws.com', 'DescribeInstances', 'Client.UnauthorizedOperation', 1)
query 2:
   ('2026-10-09T08:04:00Z', 'agent-tool-deployer', 'AttachRolePolicy', '{"roleName": "agent-tool-deployer", "policyArn": "arn:aws:iam::aws:policy/AdministratorAccess"}')
query 3:
   ('2026-10-09T08:03:00Z', 'agent-tool-deployer', 'lambda.amazonaws.com', 'CreateFunction20150331', 'arn:aws:iam::111122223333:role/AdminRole')

All three findings come out: two S3 denials and one EC2 denial for the reader role, the deployer role attaching AdministratorAccess to itself, and a Lambda function created with AdminRole. The ci-deploy role’s normal IAM change is correctly ignored, because it isn’t an agent role. Before relying on these queries, run them once in Athena against a day of real logs; translation can hide dialect differences.

Step 3: Alert in real time

Daily queries find patterns; some calls should page someone straight away. EventBridge receives CloudTrail management events as AWS API Call via CloudTrail, as long as you have a trail with logging on. A rule in the default ENABLED state doesn’t match read-only events, which is fine here, since every call in the pattern is a write. IAM is a global service whose events are recorded in US East (N. Virginia), so create this rule there:

{
  "source": ["aws.iam"],
  "detail-type": ["AWS API Call via CloudTrail"],
  "detail": {
    "eventName": ["AttachRolePolicy", "PutRolePolicy", "CreateAccessKey", "CreateUser", "UpdateAssumeRolePolicy"],
    "userIdentity": {"sessionContext": {"sessionIssuer": {"userName": [{"prefix": "agent-tool-"}]}}}
  }
}

For denied calls, a count matters more than any single event. If the trail also sends events to a CloudWatch Logs log group, a metric filter can count denials by agent roles. CloudWatch filter patterns support regular expressions between % signs, including ^ and |:

{ ($.userIdentity.sessionContext.sessionIssuer.userName = %^agent-tool-%) && ($.errorCode = %AccessDenied|UnauthorizedOperation%) }

Attach a CloudWatch alarm to the resulting metric. The same log group can be searched with CloudWatch Logs Insights, which discovers the JSON fields automatically:

fields @timestamp, eventSource, eventName, errorCode, userIdentity.sessionContext.sessionIssuer.userName
| filter userIdentity.sessionContext.sessionIssuer.userName like /^agent-tool-/
| filter errorCode like /AccessDenied|UnauthorizedOperation/
| stats count(*) as n by userIdentity.sessionContext.sessionIssuer.userName, eventSource, eventName, errorCode
| sort n desc

Note that CloudWatch Logs and EventBridge have a 256 KB maximum event size; the CloudTrail docs warn that some services emit larger events, which won’t reach these destinations. Athena over S3 still sees them.

If you already use CloudTrail Lake

Existing Lake customers can keep running queries. Two details from the documentation matter for planning:

• New accounts: an organization event data store keeps covering accounts added to your organization. Account-level event data stores don’t extend to new member accounts, so those accounts need an organization event data store or another destination.
• Migration: AWS recommends Amazon CloudWatch and supports importing historical event data stores into it. Its guide suggests a pilot with a small subset of data first, to check the schema, estimate cost from a 24-hour sample and test your queries. Data from before 2023 isn’t migrated, so if you need older events you have to keep querying them in Lake.

Lake queries are Trino SQL, and the FROM clause takes the event data store ID (the last part of its ARN). Earlier versions of this post used a made-up alias in FROM, which won’t run. Constrain eventTime in every query, because Lake charges by data scanned:

-- CloudTrail Lake (existing customers): FROM takes the event data store ID, not a name or alias.
SELECT userIdentity.sessionContext.sessionIssuer.userName AS role_name,
       eventSource, eventName, errorCode, count(*) AS n
FROM a1b2c3d4-5678-90ab-cdef-EXAMPLE11111
WHERE eventTime > '2026-10-03 00:00:00' AND eventTime < '2026-10-10 00:00:00'
  AND userIdentity.sessionContext.sessionIssuer.userName LIKE 'agent-tool-%'
  AND (errorCode LIKE '%AccessDenied%' OR errorCode LIKE '%UnauthorizedOperation%')
GROUP BY 1, 2, 3, 4
ORDER BY n DESC

The StartQuery API runs the same statement from a scheduled Lambda function; then poll GetQueryResults with the returned query ID.

Worked example: a deploy tool that grants itself admin

1. At 08:04 the EventBridge rule fires: agent-tool-deployer called AttachRolePolicy with AdministratorAccess on its own role.
2. The on-call engineer turns off the deploy tool and detaches the policy.
3. Query 3 for the same window shows that a minute earlier the role created a Lambda function with AdminRole. That was only possible because its policy allowed iam:PassRole on any role.
4. The session name in userIdentity.arn leads to the agent run in the application logs, where the tool’s arguments show the agent was working around a failed deploy.
5. The fix: restrict iam:PassRole to one execution role by ARN with an iam:PassedToService condition, deny IAM write actions for agent-tool-* roles, and add the incident as a test case.

Troubleshooting

Checklist

• ☐ Agent roles share a prefix; session names identify the agent run
• ☐ Organization trail to S3; Athena table with partition projection
• ☐ Daily queries for denials, IAM writes and passed roles
• ☐ EventBridge rule in US East (N. Virginia) for IAM writes by agent roles
• ☐ Metric filter and alarm for denied calls
• ☐ If on Lake: organization event data store, and a dated migration plan
• ☐ Every finding ends in an IAM change and a test case

FAQ

Can I still create a CloudTrail Lake event data store?

Not as a new customer since May 31, 2026. Existing customers can continue to use it.

Is Athena or CloudWatch the better replacement?

AWS’s migration guidance points Lake customers to CloudWatch. If you’re starting fresh, Athena over a trail’s S3 bucket is a simple, well-documented way to run SQL hunting queries, and CloudWatch handles alerting. Many teams use both.

Do I need data events?

Only if you need object-level activity, such as which S3 objects an agent read. They add cost, so enable them for specific buckets.

Will this stop an agent from abusing IAM?

No. It tells you quickly when it happens. Prevention is least-privilege roles, narrow tool arguments and confirmation for risky actions.

Official documentation

• CloudTrail Lake availability change and migration options
• Athena: CloudTrail table with partition projection
• CloudTrail userIdentity element
• IAM PassRole and CloudTrail
• EventBridge events from CloudTrail
• CloudWatch Logs filter pattern syntax
• CloudTrail Lake SQL constraints

Last updated on October 10, 2026