LLM Cost Controls on Bedrock: Token Budgets per PR and per Engineer

An engineer starts an agentic refactor before lunch. The agent loops on a failing test, retries every step, and by the afternoon it has used more tokens than the rest of the team did all week. Nobody notices until the monthly bill, and by then the only question left is who did it.

Cost control for large language models has to happen at request time, not invoice time. That needs two things: attribution (every model call is tied to an engineer, a pull request and a service) and enforcement (a call that would exceed a budget is slowed down or refused before it runs). This guide builds both on Amazon Bedrock with AWS-native pieces and a small gateway, and explains which AWS controls are real-time and which aren’t. It merges our earlier bootcamp lesson on per-PR budgets into this guide.

⚡ TL;DR

• Route all model calls through one gateway that knows the engineer, PR and service. Stamp that identity on every call with Converse requestMetadata (up to 16 key-value pairs, used to filter invocation logs).
• Use application inference profiles with cost allocation tags to split spend by team or service in your bill. They track cost and usage; they don’t cap it.
• Enforce budgets in the gateway: estimate tokens, reserve them with a conditional DynamoDB update, call the model, then settle with the actual usage from the response.
• Soft limit (warn, comment on the PR) at 80%; hard limit (refuse) at 100%.
• AWS Budgets actions can apply an IAM policy or SCP when spend crosses a threshold, but budget data updates only up to three times a day. It’s a backstop, not the main control.

Who this is for: platform engineers who run an internal AI gateway, IDE proxy or CI agents on Bedrock. You should know basic DynamoDB and CloudWatch.

Attribution first, or you can’t budget

You can’t enforce a per-engineer budget if a call doesn’t know which engineer made it. The simplest reliable design is a single gateway (a Lambda function or a small service) that every IDE plugin, CI agent and product feature calls instead of Bedrock directly. Engineers and agents never hold Bedrock permissions themselves; only the gateway’s role can call bedrock:InvokeModel. That gives you one place to authenticate the caller, attach identity and apply budgets.

Bedrock gives you three ways to carry that identity:

• requestMetadata on Converse. A map of up to 16 string pairs (keys up to 256 characters) that you can use to filter model invocation logs. Put engineer, pr, repo and service here.
• Application inference profiles. You create a profile that points at a foundation model or a cross-Region inference profile, attach tags such as team or service, and call the model through the profile’s ARN. With cost allocation tags enabled, spend shows up by tag in your billing data. Profiles are good for coarse splits like teams and services; a profile per engineer quickly becomes unmanageable.
• The response’s usage block. Every Converse response reports inputTokens, outputTokens and, if you use prompt caching, cacheReadInputTokens and cacheWriteInputTokens. This is the number your gateway records against budgets.

# One application inference profile per service, tagged for cost allocation
aws bedrock create-inference-profile \
  --inference-profile-name ide-proxy \
  --model-source copyFrom="arn:aws:bedrock:us-east-1:123456789012:inference-profile/us.example-model-id" \
  --tags key=service,value=ide-proxy key=team,value=devex

Replace the copyFrom ARN with the foundation model or system-defined inference profile you actually use. Then activate the tag keys as cost allocation tags in the Billing console so they appear in Cost Explorer.

Step 1: Estimate before you call

To refuse a call that would break a budget, you need an estimate before sending it. The CountTokens API returns the input token count a request would use, matching what you’d be charged for that input, and it doesn’t incur charges. Not every model supports it, so keep a fallback. Output can’t be known in advance, so reserve up to the maxTokens you set on the request. That’s another reason to set maxTokens deliberately instead of using the largest value the model allows.

import boto3
from botocore.exceptions import ClientError

brt = boto3.client("bedrock-runtime")


def estimate_tokens(model_id: str, system: list, messages: list, max_tokens: int) -> int:
    """Upper bound for budgeting: counted (or approximated) input + the output cap."""
    try:
        resp = brt.count_tokens(
            modelId=model_id,
            input={"converse": {"system": system, "messages": messages}},
        )
        input_tokens = resp["inputTokens"]
    except ClientError:
        # Model or Region without CountTokens support: rough fallback of ~4 chars/token.
        chars = sum(len(b.get("text", "")) for m in messages for b in m["content"])
        chars += sum(len(b.get("text", "")) for b in system)
        input_tokens = chars // 4 + 1
    return input_tokens + max_tokens

The character-based fallback is only a rough approximation; label it as such in your dashboards. The reservation is corrected by the real usage in Step 3, so an imperfect estimate doesn’t corrupt the budget, it just makes enforcement a little early or late.

Step 2: Reserve, call, settle

The budget lives in DynamoDB: one item per engineer per day and one per pull request. A reservation is a conditional update that only succeeds if the reserved amount still fits under the limit. This avoids the classic bug where the check (“used < limit?”) and the debit are separate, so parallel agent calls all pass the check and blow through the limit together.

import os
import time
from datetime import date

import boto3
from botocore.exceptions import ClientError

table = boto3.resource("dynamodb").Table(os.environ.get("BUDGET_TABLE", "LlmBudgets"))
brt = boto3.client("bedrock-runtime")

LIMITS = {"ENG": 2_000_000, "PR": 500_000}  # example policy: tokens per engineer-day / per PR


class BudgetExceeded(Exception):
    pass


def reserve(scope: str, key: str, tokens: int) -> None:
    limit = LIMITS[scope]
    try:
        table.update_item(
            Key={"pk": f"{scope}#{key}", "sk": "BUDGET"},
            UpdateExpression="ADD usedTokens :t SET updatedAt = :now",
            ConditionExpression="attribute_not_exists(usedTokens) OR usedTokens <= :room",
            ExpressionAttributeValues={":t": tokens, ":room": limit - tokens,
                                       ":now": int(time.time())},
        )
    except ClientError as e:
        if e.response["Error"]["Code"] == "ConditionalCheckFailedException":
            raise BudgetExceeded(f"{scope} {key} over budget") from e
        raise


def settle(scope: str, key: str, delta: int) -> None:
    table.update_item(
        Key={"pk": f"{scope}#{key}", "sk": "BUDGET"},
        UpdateExpression="ADD usedTokens :d",
        ExpressionAttributeValues={":d": delta},
    )


def governed_converse(caller: dict, model_id: str, system: list, messages: list,
                      max_tokens: int = 2048) -> dict:
    est = estimate_tokens(model_id, system, messages, max_tokens)
    eng_key = f"{caller['engineer']}#{date.today().isoformat()}"
    reserve("ENG", eng_key, est)
    pr_key = caller.get("pr")
    if pr_key:
        try:
            reserve("PR", pr_key, est)
        except BudgetExceeded:
            settle("ENG", eng_key, -est)  # release the first reservation
            raise
    actual = 0
    try:
        resp = brt.converse(
            modelId=model_id,
            system=system,
            messages=messages,
            inferenceConfig={"maxTokens": max_tokens},
            requestMetadata={k: str(v) for k, v in caller.items()},  # engineer, pr, repo, service
        )
        u = resp["usage"]
        actual = (u["inputTokens"] + u["outputTokens"]
                  + u.get("cacheReadInputTokens", 0) + u.get("cacheWriteInputTokens", 0))
        return resp
    finally:
        settle("ENG", eng_key, actual - est)
        if pr_key:
            settle("PR", pr_key, actual - est)

The finally block settles even when the call fails, so a throttled or failed request gives its reservation back. Counting cached tokens here is a policy choice: cache reads are billed at a lower rate, so if your budgets are in money rather than tokens, weight each field by its price instead of summing them.

For CI agents, put the PR budget in front of the retry logic, not behind it. Agents retry, and a budget that’s only checked once per job doesn’t stop a retry loop.

Step 3: Soft limits that people actually see

A hard stop with no warning feels like an outage. Show the meter where the work happens: in the pull request. When a PR crosses 80% of its budget, post (or update) one comment so a human can decide whether to raise the limit or stop the agent.

def budget_comment(used: int, cap: int) -> str:
    pct = round(100 * used / cap)
    bar = "#" * (pct // 10) + "-" * (10 - pct // 10) if pct <= 100 else "#" * 10
    note = "Agent paused: budget exhausted." if used >= cap else "Approaching the PR budget."
    return (f"**AI agent budget** `[{bar}]` {used:,} / {cap:,} tokens ({pct}%)\n\n"
            f"{note} A maintainer can raise the limit with the `ai-budget:+50%` label.")


def enforce(used: int, cap: int, warn, stop) -> None:
    if used >= cap:
        stop()                    # fail the job with a clear message
    elif used >= int(cap * 0.8):
        warn(budget_comment(used, cap))

Separate budgets by workload. An interactive IDE session, a CI review bot and a nightly eval suite have very different shapes, and one shared budget lets the noisiest workload starve the others.

Step 4: Metrics without a cardinality bill

CloudWatch embedded metric format (EMF) lets the gateway write metrics as structured log lines. Be careful with dimensions: every dimension set you declare creates CloudWatch metrics for each combination of values, and you pay per metric. Engineers and PR numbers are high-cardinality, so don’t make them dimensions. Keep them as plain properties in the same log line, where CloudWatch Logs Insights can still group by them, and use dimensions only for low-cardinality fields such as service and model.

import json
import time


def emit_usage(service: str, model: str, caller: dict, usage: dict, est_cost_usd: float) -> None:
    print(json.dumps({
        "_aws": {
            "Timestamp": int(time.time() * 1000),
            "CloudWatchMetrics": [{
                "Namespace": "AIGateway",
                "Dimensions": [["service"], ["service", "model"]],
                "Metrics": [
                    {"Name": "InputTokens", "Unit": "Count"},
                    {"Name": "OutputTokens", "Unit": "Count"},
                    {"Name": "EstimatedCostUSD", "Unit": "None"},
                ],
            }],
        },
        "service": service,
        "model": model,
        # Properties, not dimensions: searchable in Logs Insights, no extra metrics.
        "engineer": caller.get("engineer"),
        "pr": caller.get("pr"),
        "InputTokens": usage["inputTokens"],
        "OutputTokens": usage["outputTokens"],
        "EstimatedCostUSD": est_cost_usd,
    }))

Compute EstimatedCostUSD from the current prices on the Amazon Bedrock pricing page for your model and Region, kept in configuration rather than hard-coded. Bedrock also publishes InputTokenCount, OutputTokenCount and the cache token counts per model in the AWS/Bedrock namespace; use those to reconcile your gateway’s numbers and to catch calls that bypass the gateway.

Step 5: Account-level backstops

Your gateway is the real-time control. Two AWS features sit behind it:

• AWS Budgets actions. When a budget crosses a threshold, Budgets can apply an IAM policy or a service control policy automatically or after approval. A deny on bedrock:InvokeModel for the gateway’s role is a blunt but effective org-wide fence. Budget data is updated up to three times a day, typically 8 to 12 hours apart, so this reacts in hours, not seconds.
• Service quotas. Bedrock enforces tokens-per-minute and tokens-per-day quotas per account and model. They protect capacity, not your budget. For some models, output tokens count several times against the quota (the “burndown rate”), and throttling decisions reserve input tokens plus max_tokens up front, so quota usage isn’t a stand-in for spend.

Keep a documented kill switch as well: a configuration flag that disables non-essential AI features in the gateway. Flipping a flag is faster than editing IAM during an incident.

Route cheap work to cheap paths

Budgets limit damage; routing lowers the baseline. Send drafts and classification to a smaller model, reserve the largest model for steps that need it, cache stable prompt prefixes, and move deferrable bulk work to batch inference. Make the model choice a gateway decision based on the task type, not something each client hard-codes.

Worked example: one week of a CI review bot

1. The team sets an example policy of 500,000 tokens per PR for the review bot and a daily limit per engineer for IDE use.
2. On Tuesday, a PR with a flaky test sends the bot into a retry loop. At 80% the gateway posts a PR comment; the author adds a fix and the bot finishes under budget.
3. On Wednesday, a large generated-code PR hits 100%. The next reservation fails, the job stops with “budget exhausted”, and a maintainer adds the label to raise the limit after checking the diff.
4. On Friday, the cost report by service tag shows the review bot’s share. Logs Insights grouped by the pr property shows which PRs drove it.

Troubleshooting

Production checklist

• ☐ All model calls go through the gateway; only its role can invoke Bedrock
• ☐ requestMetadata with engineer, PR, repo and service on every call
• ☐ Application inference profiles per service, with activated cost allocation tags
• ☐ Reserve-and-settle budgets per engineer-day and per PR; separate budgets per workload
• ☐ 80% soft warning in the PR, 100% hard stop with a clear message and an override path
• ☐ EMF metrics dimensioned by service and model only; engineer and PR as properties
• ☐ Budgets action and kill switch documented and tested
• ☐ Prices kept in configuration and reviewed when models change

FAQ

Can Bedrock itself cap spend per engineer?

Not directly. Quotas are per account and model, and application inference profiles track cost rather than limit it. Per-engineer limits belong in your gateway.

Should budgets be in tokens or money?

Money is what you care about, but tokens are what you can measure per call. Many teams store tokens per model and convert with current prices for reports and limits.

Do cached tokens count?

They’re billed at the cache-read rate, which differs from normal input. Count them separately and weight by price if your budget is in money.

What about calls that bypass the gateway?

Compare the AWS/Bedrock token metrics with your gateway’s totals. A gap means someone has direct Bedrock access that should be removed.

Official documentation

• Converse API reference (requestMetadata, usage)
• Inference profiles (application profiles, cost tags)
• Count tokens before running inference
• How tokens are counted in Amazon Bedrock (burndown rates)
• Embedded metric format specification
• Configuring AWS Budgets actions

Related on CheatCoders: Bedrock prompt caching and batch inference · Overnight refactors with batch inference · DynamoDB transactions for tool ledgers

Last updated on October 10, 2026