AWS Secrets Manager Rotation for Coding-Agent Tool Credentials: Short-Lived Tokens That Can’t Leak Forever

17 views

A coding agent’s tools need credentials: a token to open pull requests, a registry token to publish packages, a database password for a migration check. The usual shortcut is an environment variable set months ago. From there the value tends to spread. It’s printed by a verbose curl into a tool result, which the model reads and may repeat; it’s captured in a trace; it’s copied into a support ticket. Rotation doesn’t stop a leak, but it limits how long a leaked value works, and fetching at call time keeps the value out of places it doesn’t need to be.

⚡ TL;DR
  • Store tool credentials in Secrets Manager and fetch them when the tool runs. The model only ever sees the tool’s name and result, never the value.
  • Pass the secret to the tool’s child process only, and redact it from the output before the output reaches the model.
  • Rotate with a Lambda function that implements Secrets Manager’s four steps: createSecret, setSecret, testSecret, finishSecret. Secrets Manager can rotate as often as every four hours.
  • For GitHub, prefer a GitHub App: store the App’s private key and mint installation tokens that expire after one hour, scoped to one repository and the permissions the tool needs.
  • For RDS, let RDS manage the master password in Secrets Manager; it rotates every seven days by default.

Who this is for: engineers running agent tools on AWS (Lambda, ECS or EC2) that call third-party or internal APIs. The rotation function was tested against moto’s Secrets Manager mock, the GitHub token code against a locally generated RSA key, and the tool runner with a real subprocess.

Where tool secrets leak, and what limits each leak

PatternHow long a leaked value worksCan the model see it?
Token in the system promptUntil someone notices and replaces itYes, always
Plaintext Lambda environment variableUntil redeploy with a new valueIf any tool prints its environment
Secrets Manager, fetched at call time, no rotationUntil manual rotationOnly if a tool echoes it (redact output)
Secrets Manager with scheduled rotationUp to the rotation interval (plus overlap)Only if echoed (redact output)
GitHub App installation tokenAt most one hourOnly if echoed (redact output)
Figure 1. Credentials flow to the tool's process, never to the model. Rotation replaces the stored value on a schedule, and GitHub tokens are minted per call from an App key. Diagram by CheatCoders.
Figure 1. Credentials flow to the tool’s process, never to the model. Rotation replaces the stored value on a schedule, and GitHub tokens are minted per call from an App key. Diagram by CheatCoders.

Step 1: Fetch at call time, and keep the value out of model context

Store each credential as a secret, named by environment, tool and (if needed) tenant, for example agents/prod/tools/registry/publish-token. The tool runner fetches it when the tool is called and passes it only to the process that needs it:

python
"""Tool runner: the secret goes into the child process only, and never back into model context."""
import subprocess


def run_with_secret(cmd: list[str], secret: str, env_name: str, cwd: str = ".") -> dict:
    proc = subprocess.run(cmd, cwd=cwd, env={"PATH": "/usr/bin:/bin", env_name: secret},
                          capture_output=True, text=True, timeout=120)
    out = (proc.stdout + proc.stderr).replace(secret, f"[{env_name} redacted]")   # e.g. verbose curl output
    return {"exit_code": proc.returncode, "output": out[-4000:]}                 # only this reaches the model


if __name__ == "__main__":
    leaky = ["sh", "-c", 'echo "Authorization: Bearer $NPM_TOKEN"; exit 3']
    print(run_with_secret(leaky, "npm_EXAMPLEexampleEXAMPLE", "NPM_TOKEN"))
text
{'exit_code': 3, 'output': 'Authorization: Bearer [NPM_TOKEN redacted]\n'}

The child process gets a minimal environment containing just the one secret, not the runner’s whole environment. Whatever the command prints, the exact secret value is replaced before the output becomes a tool result. That covers the most common leak: a verbose HTTP client printing its Authorization header. It won’t catch a transformed value (for example base64), so keep commands quiet as well. For broader detection of secrets in prompts and outputs, see Secret-Aware Context Filters.

On Lambda, the AWS Parameters and Secrets Lambda Extension can retrieve and cache secrets without SDK calls in your code. If you cache (with the extension or in your own code), keep the cache lifetime shorter than the period during which a rotated-out token still works, which Step 2 sets up.

Limit which secrets each role can read

For multi-tenant agents, tag secrets with the tenant and use attribute-based access control. The Secrets Manager documentation shows a condition that compares aws:ResourceTag/<key> on the secret with aws:PrincipalTag/<key> on the caller, so a role tagged for tenant A can’t read tenant B’s secret even if a tool asks for the wrong name.

Step 2: Rotate with the four-step Lambda function

Secrets Manager rotation calls your Lambda function four times per rotation, with a Step of createSecret, setSecret, testSecret and finishSecret, and a ClientRequestToken that is the new version’s ID. The new value is stored with the staging label AWSPENDING, tested, and then AWSCURRENT is moved to it. Each step has to be safe to run twice, because Secrets Manager can retry.

python
"""Secrets Manager rotation function for an API token that the issuing service mints and revokes."""
import json
import boto3

sm = boto3.client("secretsmanager")


class Issuer:
    """Adapter for the service that issues tokens (a package registry, an internal API...)."""
    def mint(self) -> dict: raise NotImplementedError
    def works(self, token: str) -> bool: raise NotImplementedError
    def revoke(self, token_id: str) -> None: raise NotImplementedError


def handler(event, context, issuer: Issuer):
    arn, token, step = event["SecretId"], event["ClientRequestToken"], event["Step"]
    meta = sm.describe_secret(SecretId=arn)
    if not meta.get("RotationEnabled"):
        raise ValueError(f"rotation is not enabled for {arn}")
    stages = meta["VersionIdsToStages"].get(token)
    if not stages:
        raise ValueError(f"version {token} has no stage")
    if "AWSCURRENT" in stages:
        return                                                    # already finished: retries are safe
    if "AWSPENDING" not in stages:
        raise ValueError(f"version {token} is not AWSPENDING")
    {"createSecret": create, "setSecret": lambda *a: None,       # the issuer already activated it
     "testSecret": test, "finishSecret": finish}[step](arn, token, issuer, meta)


def create(arn, token, issuer, meta):
    try:
        sm.get_secret_value(SecretId=arn, VersionId=token, VersionStage="AWSPENDING")
        return                                                    # created on an earlier attempt
    except sm.exceptions.ResourceNotFoundException:
        pass
    new = issuer.mint()                                           # {"token_id": ..., "token": ...}
    sm.put_secret_value(SecretId=arn, ClientRequestToken=token,
                        SecretString=json.dumps(new), VersionStages=["AWSPENDING"])


def test(arn, token, issuer, meta):
    pending = json.loads(sm.get_secret_value(SecretId=arn, VersionId=token,
                                             VersionStage="AWSPENDING")["SecretString"])
    if not issuer.works(pending["token"]):
        raise RuntimeError("new token failed its test call; AWSCURRENT is unchanged")


def finish(arn, token, issuer, meta):
    current = next(v for v, s in meta["VersionIdsToStages"].items() if "AWSCURRENT" in s)
    previous = next((v for v, s in meta["VersionIdsToStages"].items() if "AWSPREVIOUS" in s), None)
    sm.update_secret_version_stage(SecretId=arn, VersionStage="AWSCURRENT",
                                   MoveToVersionId=token, RemoveFromVersionId=current)
    # Revoke the token from two rotations ago. The one just demoted stays valid for one more
    # cycle, so callers holding a cached copy keep working until their cache expires.
    if previous and previous != current:
        old = json.loads(sm.get_secret_value(SecretId=arn, VersionId=previous)["SecretString"])
        issuer.revoke(old["token_id"])

Two design choices matter for agent tools:

  • Test before switching. If the new token fails its test call, the step raises, and AWSCURRENT still points to the working token. A broken rotation then shows up as a failed rotation, not as tools failing at 3 a.m.
  • Overlap by one cycle. When AWSCURRENT moves, the old version gets AWSPREVIOUS. The function revokes the token from two rotations ago, not the one it just replaced, so a tool that fetched and cached the previous value a minute ago keeps working.

The test runs three rotations against moto’s Secrets Manager mock, with a fake registry that tracks which tokens are live. Each rotation also calls finishSecret a second time, to check retries are harmless:

python
import json, uuid
import boto3
from moto import mock_aws

with mock_aws():
    import rotate_token
    rotate_token.sm = boto3.client("secretsmanager", region_name="us-east-1")
    sm = rotate_token.sm

    class FakeRegistry(rotate_token.Issuer):
        def __init__(self): self.live, self.n = set(), 0
        def mint(self):
            self.n += 1; tid = f"tok-{self.n}"; self.live.add(tid)
            return {"token_id": tid, "token": f"secret-value-{self.n}"}
        def works(self, token): return f"tok-{token.rsplit('-', 1)[1]}" in self.live
        def revoke(self, token_id): self.live.discard(token_id)

    reg = FakeRegistry()
    first = reg.mint()
    arn = sm.create_secret(Name="agents/prod/tools/registry/publish-token", SecretString=json.dumps(first))["ARN"]
    real_describe = sm.describe_secret          # test shim: moto needs a real Lambda to enable rotation
    sm.describe_secret = lambda **kw: {**real_describe(**kw), "RotationEnabled": True}

    def rotate():
        token = str(uuid.uuid4())
        # Stand-in for the service: RotateSecret registers an AWSPENDING version, then calls the steps.
        rotate_token.create(arn, token, reg, None)
        for step in ("createSecret", "setSecret", "testSecret", "finishSecret"):
            rotate_token.handler({"SecretId": arn, "ClientRequestToken": token, "Step": step}, None, reg)
        rotate_token.handler({"SecretId": arn, "ClientRequestToken": token, "Step": "finishSecret"}, None, reg)  # retry
        cur = json.loads(sm.get_secret_value(SecretId=arn, VersionStage="AWSCURRENT")["SecretString"])
        print(f"current={cur['token_id']} live={sorted(reg.live)}")

    rotate(); rotate(); rotate()
text
current=tok-2 live=['tok-1', 'tok-2']
current=tok-3 live=['tok-2', 'tok-3']
current=tok-4 live=['tok-3', 'tok-4']

After each rotation exactly two tokens are live, the current one and the previous one, and the token from two rotations ago has been revoked. The test uses a small shim because moto requires a real Lambda function before it marks rotation as enabled; in AWS you attach the function with RotateSecret and a schedule. Secrets Manager can rotate as often as every four hours, within a rotation window of at least one hour.

Step 3: Use GitHub App tokens instead of personal access tokens

For GitHub there’s a better option than rotating a long-lived token. A GitHub App’s installation access tokens expire after one hour, and when you request one you can limit it to specific repositories (the repositories body parameter) and to a subset of the App’s permissions (the permissions parameter). Secrets Manager then stores only the App’s private key and client ID.

To request a token, sign a JSON Web Token with the App’s private key. GitHub requires RS256, recommends setting iat 60 seconds in the past to allow for clock drift, limits exp to 10 minutes ahead, and recommends the client ID for iss. Then POST to /app/installations/{installation_id}/access_tokens:

python
"""Mint a short-lived GitHub App installation token per tool call. Only the App key lives in Secrets Manager."""
import calendar
import json
import time
import urllib.request

import boto3
import jwt  # PyJWT, with the cryptography package for RS256

sm = boto3.client("secretsmanager")
SECRET_ID = "agents/prod/tools/github/app"   # {"client_id": "...", "private_key": "-----BEGIN ..."}
_cache: dict[tuple, tuple[str, float]] = {}


def app_jwt(client_id: str, private_key: str, now: float | None = None) -> str:
    now = int(now or time.time())
    # GitHub: RS256, iat 60 s in the past for clock drift, exp no more than 10 minutes ahead.
    return jwt.encode({"iat": now - 60, "exp": now + 540, "iss": client_id}, private_key, algorithm="RS256")


def installation_token(installation_id: int, repo: str, permissions: dict, post=None) -> str:
    key = (installation_id, repo, json.dumps(permissions, sort_keys=True))
    if key in _cache and _cache[key][1] - time.time() > 300:     # reuse until 5 minutes before expiry
        return _cache[key][0]
    app = json.loads(sm.get_secret_value(SecretId=SECRET_ID)["SecretString"])
    body = {"repositories": [repo], "permissions": permissions}   # narrower than the App's own grants
    resp = (post or _post)(f"https://api.github.com/app/installations/{installation_id}/access_tokens",
                           app_jwt(app["client_id"], app["private_key"]), body)
    expires = calendar.timegm(time.strptime(resp["expires_at"], "%Y-%m-%dT%H:%M:%SZ"))   # UTC
    _cache[key] = (resp["token"], expires)
    return resp["token"]


def _post(url: str, bearer: str, body: dict) -> dict:
    req = urllib.request.Request(url, data=json.dumps(body).encode(), method="POST", headers={
        "Accept": "application/vnd.github+json", "Authorization": f"Bearer {bearer}"})
    with urllib.request.urlopen(req, timeout=10) as r:
        return json.load(r)
python
import json, time
import boto3, jwt
from cryptography.hazmat.primitives import serialization
from cryptography.hazmat.primitives.asymmetric import rsa
from moto import mock_aws

key = rsa.generate_private_key(public_exponent=65537, key_size=2048)
pem = key.private_bytes(serialization.Encoding.PEM, serialization.PrivateFormat.PKCS8,
                        serialization.NoEncryption()).decode()

with mock_aws():
    import github_token as gt
    gt.sm = boto3.client("secretsmanager", region_name="us-east-1")
    gt.sm.create_secret(Name=gt.SECRET_ID, SecretString=json.dumps({"client_id": "Iv23liEXAMPLE", "private_key": pem}))
    calls = []
    def fake_post(url, bearer, body):
        claims = jwt.decode(bearer, key.public_key(), algorithms=["RS256"])
        calls.append((url, body, claims["exp"] - claims["iat"]))
        expires = time.strftime("%Y-%m-%dT%H:%M:%SZ", time.gmtime(time.time() + 3600))
        return {"token": f"ghs_example{len(calls)}", "expires_at": expires}
    perms = {"contents": "write", "pull_requests": "write"}
    t1 = gt.installation_token(42, "web-app", perms, post=fake_post)
    t2 = gt.installation_token(42, "web-app", perms, post=fake_post)       # cached
    t3 = gt.installation_token(42, "web-app", {"contents": "read"}, post=fake_post)
    print(t1, t2, t3, "| API calls:", len(calls))
    print("JWT lifetime (s):", calls[0][2], "| body:", calls[0][1])
text
ghs_example1 ghs_example1 ghs_example2 | API calls: 2
JWT lifetime (s): 600 | body: {'repositories': ['web-app'], 'permissions': {'contents': 'write', 'pull_requests': 'write'}}

The test signs with a freshly generated RSA key and checks the JWT verifies with the public key. Repeated calls with the same repository and permissions reuse the cached token; a call that needs different permissions (read-only here) mints a separate, narrower token. A tool that only opens pull requests on one repository never holds a token that can push to every repository the App is installed on.

Earlier versions of this post suggested automating the creation of fine-grained personal access tokens. GitHub doesn’t document an API for creating personal access tokens, so that approach isn’t something you can build on; use a GitHub App.

Step 4: Databases

For Amazon RDS, let RDS manage the master user password in Secrets Manager. RDS creates the secret, rotates it every seven days by default (you can change the schedule), and applications read the current value from the secret. Agent tools that need database access should use a separate, least-privilege database user rather than the master user; rotate that user’s password with a rotation function in the same four-step pattern.

Worked example: a token printed into a tool result

  1. A publish tool fails, and the agent retries it with verbose output. The registry client prints its request headers, including the token.
  2. The runner’s redaction replaces the token in the output before the tool result goes to the model, so the transcript and the trace contain [NPM_TOKEN redacted].
  3. Suppose redaction had been missing. The token would now be in the conversation history and traces. Because of the one-cycle overlap, a single rotation leaves the leaked token valid as AWSPREVIOUS; it’s only revoked when it becomes two versions old. With weekly rotation that could be up to 14 days, so after a known leak revoke the token directly with the issuer (or rotate twice) instead of waiting.
  4. The follow-up fix: run the publish command without verbose flags, and add a test that a tool echoing its secret gets it redacted.

Troubleshooting

SymptomLikely causeFix
Rotation fails at testSecretNew token not active yet, or wrong scopeCheck the issuer; AWSCURRENT is unchanged, so tools keep working
Tools fail right after rotationOld token revoked while callers still cache itRevoke two versions back (as above); shorten caches
Rotation retries create extra tokenscreateSecret not idempotentCheck for an existing AWSPENDING value for the token before minting
GitHub returns 401 for the JWTClock drift or exp more than 10 minutes aheadSet iat 60 seconds back; sync the clock
Wrong tenant’s secret readRole can read every secret by nameTag-based conditions on GetSecretValue

Checklist

  • ☐ No credential values in prompts, tool schemas or plaintext environment variables
  • ☐ Secrets fetched at call time; passed to the child process only
  • ☐ Tool output redacted before it reaches the model
  • ☐ Rotation function implements all four steps, idempotently, and tests before switching
  • ☐ Previous token stays valid for one cycle; caches are shorter than that
  • ☐ GitHub access through an App with per-call, per-repository tokens
  • ☐ Tag-based access so roles only read their tenant’s secrets

FAQ

Isn’t Parameter Store enough?

Parameter Store SecureString parameters are fine for configuration and values you rotate yourself. Secrets Manager adds managed rotation with staging labels, which is what this pattern relies on.

How often should I rotate?

As often as the issuing service and your callers can handle. Secrets Manager supports every four hours at most; weekly is a common start for tokens. GitHub App tokens make the question moot for GitHub.

Can the agent ever be given a secret value?

There’s rarely a reason. If a task needs a credential, it belongs in a tool that uses it, not in the conversation.

Does rotation help after a leak?

It limits how long the leaked value works. After a known leak, revoke the leaked token with the issuer right away; with the overlap described above, one rotation alone leaves it valid for another cycle.

Official documentation

Last updated on October 10, 2026

Watch: 100 Days of AI on YouTube

Short videos from CheatCoders, one AI engineering topic per day.

Open the playlist on YouTube · CheatCoders on YouTube

Deep-dive PDF

Get the expanded guide for this post — extra diagrams-style checklists, failure modes, and a production walkthrough. Free when you subscribe to CheatCoders.

Already subscribed? or open the subscribe page.


Discover more from CheatCoders

Subscribe to get the latest posts sent to your email.

1 Comment

  1. From what I’ve seen, an SEO consultant is kind of like a doctor for your website. They diagnose problems like slow pages, missing meta tags, or weak content, then prescribe fixes based on data instead of guesswork. The good ones also keep an eye on algorithm updates so your rankings don’t randomly tank overnight, which happened to me once before I hired someone.

    I guess the short version is they make sure Google and actual users can find and trust your site, through a mix of technical tweaks, content advice, and link building. It’s not an overnight fix though, mine warned me it’d take months before seeing real movement, and he was right. Has anyone had a consultant who delivered faster results, or is that just unrealistic?

    Been doing my own SEO for years but finally brought in outside help when traffic plateaued. What surprised me was how much time he spent asking about my business goals before even touching keywords. Turns out half my problem wasn’t technical, it was just targeting the wrong audience the whole time.

    If I could go back I’d ask way more questions before signing anything, like how they measure success and what tools they actually use. A lot of this industry runs on jargon that sounds impressive but means nothing in practice. What questions do you guys usually ask before trusting someone with your site?
    hire an SEO consultant [url=https://yourhomeyourwayltd.co.uk/what-an-seo-consultant-actually-does-advice/#hire-an-SEO-consultant]https://yourhomeyourwayltd.co.uk/what-an-seo-consultant-actually-does-advice/[/url] .

Leave a comment

No account needed. Name and email are optional.