AWS Budgets + Cost Anomaly Detection: Cap Runaway Coding-Agent Spend

One buggy planner loop, one missing recursive loop protection, or one tenant who finds an unbounded run_tests tool — and your Bedrock bill becomes the postmortem. CloudWatch EMF per-tenant tokens (EMF LLM cost) tell you who; AWS Budgets and Cost Anomaly Detection tell finance when to page and give you SNS hooks to flip kill switches. This post wires both for coding-agent platforms without waiting for month-end surprise.

⚡ TL;DR: Create monthly/daily Budgets on services agents touch (Bedrock, Lambda, Batch, OpenSearch) with 50/80/100% SNS alerts. Enable Cost Anomaly Detection on those services. On alarm, Lambda sets AppConfig kill switches and optionally disables API keys. Pair with EMF + DynamoDB quota ledgers for soft real-time caps. Related: EMF per-tenant tokens, AppConfig kill switches, API Gateway WAF rate limits.

Defense in depth for agent spend

Budgets are not a substitute for DynamoDB token ledgers — billing data lags. Use Budgets as the backstop when soft caps fail.

Tag everything agents touch

Cost allocation tags make Budgets useful:

Application = coding-agents
TenantId    = acme          (when possible on resources)
Env         = prod
Tool        = typecheck     (on dedicated functions)

Activate tags in Billing → Cost allocation tags. For Bedrock, also emit EMF with TenantId / ModelId dimensions so engineering can act before CUR catches up.

# ❌ Untagged shared account dump — Budget fires, nobody knows which agent
# ✅ Separate linked account or strict tags for agent workloads

Create Budgets that page humans and bots

# Sketch: monthly Bedrock+Lambda budget with SNS
aws budgets create-budget --account-id 123456789012 --budget '{
  "BudgetName": "coding-agents-monthly",
  "BudgetType": "COST",
  "TimeUnit": "MONTHLY",
  "BudgetLimit": {"Amount": "2500", "Unit": "USD"},
  "CostFilters": {
    "TagKeyValue": ["user:Application$coding-agents"]
  },
  "CostTypes": {
    "IncludeTax": true,
    "IncludeSubscription": true,
    "UseBlended": false
  }
}' --notifications-with-subscribers '[
  {
    "Notification": {
      "NotificationType": "ACTUAL",
      "ComparisonOperator": "GREATER_THAN",
      "Threshold": 50,
      "ThresholdType": "PERCENTAGE"
    },
    "Subscribers": [{"SubscriptionType": "SNS", "Address": "arn:aws:sns:us-east-1:123456789012:agent-cost-alerts"}]
  },
  {
    "Notification": {
      "NotificationType": "ACTUAL",
      "ComparisonOperator": "GREATER_THAN",
      "Threshold": 80,
      "ThresholdType": "PERCENTAGE"
    },
    "Subscribers": [{"SubscriptionType": "SNS", "Address": "arn:aws:sns:us-east-1:123456789012:agent-cost-alerts"}]
  }
]'

Add a daily Budget at ~1/20 of monthly for faster signal on runaway nights (Batch jobs, Scheduler overnight batches). Forecasted notifications help when month-to-date velocity projects overage.

Cost Anomaly Detection for unknown-unknowns

Budgets catch thresholds you chose. Anomalies catch “Bedrock spend 6× vs last week though still under monthly cap.”

1. Create a cost monitor on linked account or services: Amazon Bedrock, AWS Lambda, AWS Batch, Amazon OpenSearch Service.
2. Subscribe SNS / email.
3. On anomaly, same kill-switch Lambda as Budgets.

// ✅ SNS → Lambda: flip AppConfig kill switch for all non-essential tools
import {
  AppConfigDataClient,
  // use AppConfig start-configuration-session / or prefer Admin API to update hosted config
} from "@aws-sdk/client-appconfig";

export async function onCostAlarm(event: { severity: "warn" | "critical" }) {
  // Pseudocode: write flag file / update hosted configuration
  // agents.tools.allowlist = ["health", "retrieve_only"] on critical
  console.log(JSON.stringify({ msg: "cost_kill_switch", severity: event.severity }));
  // Also post Slack; open incident ticket
}

Prefer updating a hosted configuration document that tool gateways already poll (AppConfig kill switches) over redeploying thirty Lambdas at 2am.

Soft real-time caps still required

Wire DynamoDB quota (see today’s ledger post pattern) + EMF alarms:

// EMF-style metric log for Bedrock tokens (sketch)
console.log(
  JSON.stringify({
    _aws: {
      Timestamp: Date.now(),
      CloudWatchMetrics: [
        {
          Namespace: "CodingAgents/LLM",
          Dimensions: [["TenantId", "ModelId"]],
          Metrics: [
            { Name: "TokensIn", Unit: "Count" },
            { Name: "TokensOut", Unit: "Count" },
            { Name: "EstimatedCostUsd", Unit: "None" },
          ],
        },
      ],
    },
    TenantId: tenantId,
    ModelId: modelId,
    TokensIn: tokensIn,
    TokensOut: tokensOut,
    EstimatedCostUsd: estimateUsd(tokensIn, tokensOut, modelId),
  })
);

Alarm when EstimatedCostUsd sum per tenant exceeds daily soft cap → deny new sessions for that tenant while Budgets watch the account.

Production checklist

• [ ] Cost allocation tags on all agent resources; activated in billing.
• [ ] Monthly + daily Budgets on tagged agent spend; SNS to eng + pager.
• [ ] Cost Anomaly Detection monitors on Bedrock/Lambda/Batch/OpenSearch.
• [ ] SNS consumers flip AppConfig kill switches (tested quarterly).
• [ ] EMF + DynamoDB quotas for per-tenant real-time soft caps.
• [ ] WAF/usage plans on public agent endpoints.
• [ ] Runbook: who acknowledges, how to restore tools, finance CC list.
• [ ] Separate linked account for prod agents if blast radius demands it.

Budgets and anomaly detection will not stop a tight loop in 30 seconds — your ledger and WAF must. They will stop a quiet week of 4× Bedrock spend from becoming a board-level surprise. Cap the agent before it caps your runway.

Worked example: night Batch spike

Scenario: EventBridge Scheduler kicks off overnight refactors (Scheduler + Batch). A bad job definition retries forever on Spot reclaim.

1. 00:40 EMF shows tenant acme tokens climbing; soft DynamoDB quota not applied to Batch path (gap).
2. 01:10 Daily Budget 80% SNS fires → Lambda sets AppConfig batch_overnight=false.
3. 01:12 Scheduler targets check flag / EventBridge rule disabled via same automation.
4. 01:30 Cost Anomaly Detection emails “Batch +320%”; already contained.
5. Postmortem: extend ledger quotas to Batch job enqueue; add maxRetry on job def; Budget threshold lowered for daily.

Without Budgets+SNS automation, you discover this in the CUR on Friday.

What not to do

• ❌ Single annual Budget at $50k with email-only to finance.
• ❌ Kill the entire AWS account with a Service Control Policy on first warn.
• ❌ Trust Budgets alone without per-tenant soft caps.
• ❌ Ignore Marketplace/third-party model bills outside Bedrock filters.
• ❌ Leave SNS topic without publisher restriction (confused-deputy noise).

Cap runaways in layers. Budgets and anomaly detection are the seatbelt after the DynamoDB quota brakes.

Mapping alerts to concrete AppConfig flags

Keep a small, documented flag schema so on-call does not invent names at 2am:

Gateway and tool runners must fail closed when AppConfig is unreachable after a cost event (cached last-good with short TTL is fine for availability — but after kill-switch write, force refresh). Combine with WAF rate limits so public traffic cannot bypass by hammering until flags propagate.