12 views
One buggy planner loop, one missing recursive loop protection, or one tenant who finds an unbounded run_tests tool — and your Bedrock bill becomes the postmortem. CloudWatch EMF per-tenant tokens (EMF LLM cost) tell you who; AWS Budgets and Cost Anomaly Detection tell finance when to page and give you SNS hooks to flip kill switches. This post wires both for coding-agent platforms without waiting for month-end surprise.
⚡ TL;DR: Create monthly/daily Budgets on services agents touch (Bedrock, Lambda, Batch, OpenSearch) with 50/80/100% SNS alerts. Enable Cost Anomaly Detection on those services. On alarm, Lambda sets AppConfig kill switches and optionally disables API keys. Pair with EMF + DynamoDB quota ledgers for soft real-time caps. Related: EMF per-tenant tokens, AppConfig kill switches, API Gateway WAF rate limits.
Defense in depth for agent spend
| Layer | Mechanism | Latency | Granularity |
|---|---|---|---|
| Tool ledger quota | DynamoDB remaining_tokens | Real-time | Tenant / month |
| EMF + alarms | Custom metrics | Near real-time | Tenant / model |
| API WAF / usage plans | Request caps | Real-time | API key |
| AWS Budgets | $ thresholds | Hours (billing) | Account / service / tag |
| Cost Anomaly Detection | ML on spend | Hours | Service / account |
Tag everything agents touch
Cost allocation tags make Budgets useful:text
Activate tags in Billing → Cost allocation tags. For Bedrock, also emit EMF with Application = coding-agents
TenantId = acme (when possible on resources)
Env = prod
Tool = typecheck (on dedicated functions)
TenantId / ModelId dimensions so engineering can act before CUR catches up.
bash
# ❌ Untagged shared account dump — Budget fires, nobody knows which agent
# ✅ Separate linked account or strict tags for agent workloads
Create Budgets that page humans and bots
bash
Add a daily Budget at ~1/20 of monthly for faster signal on runaway nights (Batch jobs, Scheduler overnight batches). Forecasted notifications help when month-to-date velocity projects overage.
# Sketch: monthly Bedrock+Lambda budget with SNS
aws budgets create-budget --account-id 123456789012 --budget '{
"BudgetName": "coding-agents-monthly",
"BudgetType": "COST",
"TimeUnit": "MONTHLY",
"BudgetLimit": {"Amount": "2500", "Unit": "USD"},
"CostFilters": {
"TagKeyValue": ["user:Application$coding-agents"]
},
"CostTypes": {
"IncludeTax": true,
"IncludeSubscription": true,
"UseBlended": false
}
}' --notifications-with-subscribers '[
{
"Notification": {
"NotificationType": "ACTUAL",
"ComparisonOperator": "GREATER_THAN",
"Threshold": 50,
"ThresholdType": "PERCENTAGE"
},
"Subscribers": [{"SubscriptionType": "SNS", "Address": "arn:aws:sns:us-east-1:123456789012:agent-cost-alerts"}]
},
{
"Notification": {
"NotificationType": "ACTUAL",
"ComparisonOperator": "GREATER_THAN",
"Threshold": 80,
"ThresholdType": "PERCENTAGE"
},
"Subscribers": [{"SubscriptionType": "SNS", "Address": "arn:aws:sns:us-east-1:123456789012:agent-cost-alerts"}]
}
]'
Cost Anomaly Detection for unknown-unknowns
Budgets catch thresholds you chose. Anomalies catch “Bedrock spend 6× vs last week though still under monthly cap.”- Create a cost monitor on linked account or services:
Amazon Bedrock,AWS Lambda,AWS Batch,Amazon OpenSearch Service. - Subscribe SNS / email.
- On anomaly, same kill-switch Lambda as Budgets.
typescript
Prefer updating a hosted configuration document that tool gateways already poll (AppConfig kill switches) over redeploying thirty Lambdas at 2am.
// ✅ SNS → Lambda: flip AppConfig kill switch for all non-essential tools
import {
AppConfigDataClient,
// use AppConfig start-configuration-session / or prefer Admin API to update hosted config
} from "@aws-sdk/client-appconfig";
export async function onCostAlarm(event: { severity: "warn" | "critical" }) {
// Pseudocode: write flag file / update hosted configuration
// agents.tools.allowlist = ["health", "retrieve_only"] on critical
console.log(JSON.stringify({ msg: "cost_kill_switch", severity: event.severity }));
// Also post Slack; open incident ticket
}
Soft real-time caps still required
Wire DynamoDB quota (see today’s ledger post pattern) + EMF alarms:typescript
Alarm when // EMF-style metric log for Bedrock tokens (sketch)
console.log(
JSON.stringify({
_aws: {
Timestamp: Date.now(),
CloudWatchMetrics: [
{
Namespace: "CodingAgents/LLM",
Dimensions: [["TenantId", "ModelId"]],
Metrics: [
{ Name: "TokensIn", Unit: "Count" },
{ Name: "TokensOut", Unit: "Count" },
{ Name: "EstimatedCostUsd", Unit: "None" },
],
},
],
},
TenantId: tenantId,
ModelId: modelId,
TokensIn: tokensIn,
TokensOut: tokensOut,
EstimatedCostUsd: estimateUsd(tokensIn, tokensOut, modelId),
})
);
EstimatedCostUsd sum per tenant exceeds daily soft cap → deny new sessions for that tenant while Budgets watch the account.
Production checklist
- [ ] Cost allocation tags on all agent resources; activated in billing.
- [ ] Monthly + daily Budgets on tagged agent spend; SNS to eng + pager.
- [ ] Cost Anomaly Detection monitors on Bedrock/Lambda/Batch/OpenSearch.
- [ ] SNS consumers flip AppConfig kill switches (tested quarterly).
- [ ] EMF + DynamoDB quotas for per-tenant real-time soft caps.
- [ ] WAF/usage plans on public agent endpoints.
- [ ] Runbook: who acknowledges, how to restore tools, finance CC list.
- [ ] Separate linked account for prod agents if blast radius demands it.
Worked example: night Batch spike
Scenario: EventBridge Scheduler kicks off overnight refactors (Scheduler + Batch). A bad job definition retries forever on Spot reclaim.- 00:40 EMF shows tenant
acmetokens climbing; soft DynamoDB quota not applied to Batch path (gap). - 01:10 Daily Budget 80% SNS fires → Lambda sets AppConfig
batch_overnight=false. - 01:12 Scheduler targets check flag / EventBridge rule disabled via same automation.
- 01:30 Cost Anomaly Detection emails “Batch +320%”; already contained.
- Postmortem: extend ledger quotas to Batch job enqueue; add maxRetry on job def; Budget threshold lowered for daily.
What not to do
- ❌ Single annual Budget at $50k with email-only to finance.
- ❌ Kill the entire AWS account with a Service Control Policy on first warn.
- ❌ Trust Budgets alone without per-tenant soft caps.
- ❌ Ignore Marketplace/third-party model bills outside Bedrock filters.
- ❌ Leave SNS topic without publisher restriction (confused-deputy noise).
Mapping alerts to concrete AppConfig flags
Keep a small, documented flag schema so on-call does not invent names at 2am:| Flag | Default | On Budget 80% | On Budget 100% / critical anomaly |
|---|---|---|---|
tools.heavy_enabled |
true | false | false |
tools.batch_overnight |
true | false | false |
models.allow_opus |
true | true | false |
models.allow_haiku |
true | true | true |
ingress.public_agent |
true | true | false |
Watch: 100 Days of AI on YouTube
Short videos from CheatCoders, one AI engineering topic per day.
Most viewed
- Python Decorators Explained: From Simple Wrappers to Production Patterns
- Claude Code vs Cursor vs GitHub Copilot: Choose by Workflow, and Set Up One Repo That Works With All Three
- AI Agent Frameworks in 2025: LangGraph vs CrewAI vs AutoGen vs Raw API
- Python String Methods: Every str Method With Real Production Examples
- REST API Design Best Practices: The Patterns That Make APIs a Joy to Use
Newly added
- n8n vs AI Agent: I Gave Both the Same Job (Local, Ollama)
- Local AI Video Generator: Build Your Own App (Free, Python, No GPU)
- AI Wrote Raw Binary Instead of Code. Here’s What Happened (WebAssembly)
- I Built a Browser That Only Runs Encrypted Binary (Ep 1)
- RAG From Scratch: Chunking, Embeddings, Reranking and Evaluation (Real Numbers)
Deep-dive PDF
Get the expanded guide for this post — extra diagrams-style checklists, failure modes, and a production walkthrough. Free when you subscribe to CheatCoders.
Already subscribed? or open the subscribe page.
Discover more from CheatCoders
Subscribe to get the latest posts sent to your email.