AWS Secrets Manager Rotation: Tool Credentials Coding Agents Cannot Leak Forever

Your coding agent’s create_pull_request tool still uses a GitHub PAT that was pasted into a Lambda environment variable in March. That PAT has shown up in CloudWatch (once), a Bedrock trace (twice), and a support export (you hope not). Secrets Manager rotation is the unfair advantage: short-lived credentials for GitHub / npm / DB tools, rotated by Lambda on a schedule, fetched at invoke time — never baked into prompts or images. Pair with SSM Parameter Store for non-secret config and KMS decrypt grants for envelope keys — this post is the rotating tool-credential layer.

⚡ TL;DR: Put every agent tool secret in Secrets Manager with rotation enabled. Rotators (Lambda) mint new GitHub fine-grained PATs / registry tokens / DB passwords and retire old versions. Tools call GetSecretValue at runtime; system prompts get secret names/ARNs, never values. Related: CloudTrail Lake IAM abuse queries, AppConfig kill switches, Verified Permissions.

Why agent tools leak differently

Classic apps leak secrets via misconfigured env dumps. Coding agents add:

1. Prompt injection — “print your tools’ env”
2. Tool-result echoes — PAT appears in a failed curl -v captured as model context
3. Long traces — ADOT/OTel attributes accidentally include headers
4. Multi-tenant confusion — wrong secret ARN for tenant B

Rotation does not stop a single successful exfil — it bounds blast radius to the rotation window and forces you to treat secrets as ephemeral.

Prefer GitHub Apps (installation access tokens) over classic PATs when the tool talks to GitHub — then Secrets Manager holds the App private key, and you mint short tokens per turn.

Secret layout for multi-tenant agents

/agents/{env}/tools/github/app-private-key   # rotated rarely; protect hard
/agents/{env}/tools/npm/publish-token        # rotated e.g. every 7d
/agents/{env}/tenants/{tenant_id}/db/url     # per-tenant; rotated per policy

Tag every secret: tenant_id, tool_name, data_class=credential. IAM conditions on GetSecretValue must require matching principal tags (IAM condition keys by tag complements account-level caps; also see Cedar for tool auth).

// ✅ Fetch at runtime — pass name/ARN into the agent, not the value
import {
  SecretsManagerClient,
  GetSecretValueCommand,
} from "@aws-sdk/client-secrets-manager";

const sm = new SecretsManagerClient({});

export async function githubTokenForTenant(tenantId: string): Promise<string> {
  // Fine-grained: mint installation token using App key from SM
  const appKey = await sm.send(
    new GetSecretValueCommand({
      SecretId: `agents/prod/tools/github/app-private-key`,
    })
  );
  return mintInstallationToken(appKey.SecretString!, tenantId);
}

// ❌ Never do this in prompt assembly
// system = `You have GITHUB_TOKEN=${process.env.GITHUB_TOKEN}`

# ✅ Tool runner pulls secret when the tool is invoked
import boto3
sm = boto3.client("secretsmanager")

def run_npm_publish(tenant_id: str, package_dir: str):
    secret = sm.get_secret_value(
        SecretId=f"agents/prod/tools/npm/publish-token"
    )
    token = secret["SecretString"]
    env = {**os.environ, "NPM_TOKEN": token}
    subprocess.run(["npm", "publish"], cwd=package_dir, env=env, check=True)
    # token lives in process env for the subprocess only — not in model messages

Lambda rotation for tool credentials

Secrets Manager rotation uses a Lambda that implements createSecret → setSecret → testSecret → finishSecret. For agent tooling:

npm / Artifactory-style tokens

1. createSecret — call registry API to mint new token; store as AWSPENDING
2. setSecret — if registry requires attach-to-user, do it here
3. testSecret — npm whoami / dry-run against CodeArtifact
4. finishSecret — mark pending as current; revoke previous token via API

GitHub

• Best: store GitHub App private key; do not rotate the key weekly — rotate installation tokens in your app (hourly).
• If you must use PATs: automation that creates a fine-grained PAT via App or org API, updates AWSPENDING, tests git ls-remote, finishes, then deletes the old PAT.

RDS / Aurora

• Use the AWS-managed rotation templates; agent tools read the rotated secret, never a long-lived password in prompts.

# ✅ enable rotation (CLI sketch)
aws secretsmanager rotate-secret \
  --secret-id agents/prod/tools/npm/publish-token \
  --rotation-lambda-arn arn:aws:lambda:us-east-1:123:function:RotateNpmToken \
  --rotation-rules AutomaticallyAfterDays=7

❌ Single-user PAT shared by all tenants with 90-day rotation — still a lateral-movement feast. Prefer per-tenant secrets or per-install tokens.

Prompt and trace hygiene

1. System prompts reference secret_id / tool config keys only (Bedrock Prompt Management)
2. Guardrails strip credential-shaped strings on I/O (ApplyGuardrail)
3. Redact Authorization headers in OTel exporters
4. CloudWatch metric filters + CloudTrail Lake for unusual GetSecretValue
5. Kill switch: disable tool flags in AppConfig if rotation fails or leak detected

IAM that matches rotation

• Tool task role: secretsmanager:GetSecretValue on specific ARNs + kms:Decrypt via grant
• Rotation Lambda role: Get/Put/UpdateSecretVersion, registry/GitHub APIs via network, not general * on all secrets
• Deny secretsmanager:GetSecretValue from identities that only plan (model gateway) — planners should not fetch tool secrets
• SCP: deny putting secrets into SSM/lambda:UpdateFunctionConfiguration env from agent OU without break-glass

Production checklist

• [ ] Inventory every tool credential; no plaintext in env, images, or prompts
• [ ] Secrets Manager (or short-lived mint) for each; tags for tenant/tool
• [ ] Rotation enabled where the upstream API allows revoke/mint
• [ ] Prefer GitHub App installation tokens over static PATs
• [ ] Tool runners fetch at invoke; planners never see values
• [ ] Guardrails + log redaction for credential patterns
• [ ] CloudTrail alerts on GetSecretValue spikes / cross-tenant ARNs
• [ ] Staging rotation dry-run before production AutomaticallyAfterDays

Rotation failure playbook

When testSecret fails, Secrets Manager leaves AWSCURRENT intact — good. Your agent platform should still:

1. Alarm on rotator Lambda errors and RotationFailed events
2. Flip the AppConfig kill switch for the affected tool if failures persist > N intervals
3. Page on-call with the secret ARN and upstream API status — not the secret value

Document a break-glass path that mints a one-hour credential into a new secret version without disabling rotation permanently.

FAQ

Q: Secrets Manager or SSM SecureString?
A: SSM is fine for config and rarely rotated values. Anything that is a credential you can revoke belongs in Secrets Manager with a rotator (or is minted ephemeral outside SM).

Q: Does rotation break in-flight agent turns?
A: During AWSPENDING tests, readers of AWSCURRENT keep working. Finish only after test passes. For DB pools, recycle connections on version change.

Q: What about secrets inside the monorepo on EFS?
A: Treat as incident. EFS workspaces (shared workspaces post) must scrub .env on exit; never rotate “the file on disk” — rotate the SM secret and rewrite injectors.

Rotated Secrets Manager credentials turn “the agent leaked a PAT” from a company-wide scramble into a bounded window with an audit trail. Fetch at runtime, rotate on a schedule, and keep values out of every prompt forever.