Amazon Bedrock Model Evaluation: Score Coding-Agent Outputs Before You Promote a Prompt

You changed one paragraph in the coding-agent system prompt, traffic shifted to Claude Sonnet “because it’s newer,” and unit-test pass rate dropped 8% on Monday. Nobody can prove which change did it. Amazon Bedrock Model Evaluation is the unfair advantage: managed jobs that score candidate models/prompts against your dataset with automatic metrics (and optional human review) before Prompt Management pins a new production version. This is not Prompt Flows (orchestration) and not a one-off Athena notebook over traces — it is the promotion gate. Store eval artifacts immutably (Object Lock) and query historical scores with Athena.

⚡ TL;DR: Build a golden coding-task dataset (prompt → expected signals). Run Bedrock Model Evaluation (automatic) on baseline vs candidate prompt/model. Gate Prompt Management “prod” aliases on eval thresholds. Log scores to S3; alarm on regressions. Related: Prompt Management, Prompt Flows, Athena eval traces, Budgets.

Why vibes-based prompt deploys fail

Coding agents are multi-metric:

Bedrock Model Evaluation gives you a repeatable job over a dataset so baseline vs candidate is comparable. Pair automatic scores with a small human review sample for high-risk prompt classes (security tools, prod DDL).

Dataset design for coding agents

Put JSONL (or the console-supported format) in S3:

{"prompt": "Fix the off-by-one in binary_search.py. Return only a unified diff.", "category": "bugfix", "reference": "expected signals: mid=(lo+hi)//2 fix, tests mentioned"}
{"prompt": "Add retries with jitter to fetchUser. Do not break types.", "category": "refactor", "reference": "exponential backoff, TypeScript types preserved"}

Cover categories your agent actually serves: bugfix, refactor, test-gen, explain, dangerous-tool refusal. Include prompt-injection cases that must refuse. Size: start at 50–200 tasks; grow to thousands as you automate.

# ✅ upload eval dataset
import boto3, json
s3 = boto3.client("s3")
bucket = "coding-agent-eval-prod"
rows = [...]  # list[dict]
body = "\n".join(json.dumps(r) for r in rows).encode()
s3.put_object(Bucket=bucket, Key="datasets/coding-v3/data.jsonl", Body=body)

Create an automatic evaluation job

Console or API: choose models (or the same model with different inference configs / prompts), attach dataset, select metrics suitable for generation quality, and write results to S3.

# ✅ illustrative — create model evaluation job (API shape evolves; verify in current AWS docs)
aws bedrock create-evaluation-job \
  --job-name "coding-agent-prompt-v12-vs-v11" \
  --role-arn arn:aws:iam::111122223333:role/BedrockEvalRole \
  --evaluation-config file://eval-config.json \
  --inference-config file://inference-config.json \
  --output-data-config s3Bucket=coding-agent-eval-prod,s3Prefix=results/2026-09-26/ \
  --job-tags Key=workload,Value=coding-agent

// ✅ promotion gate in CI — fail if candidate score < baseline - epsilon
type EvalSummary = { jobName: string; avgScore: number; safetyFailRate: number };

export function shouldPromote(baseline: EvalSummary, candidate: EvalSummary): boolean {
  if (candidate.safetyFailRate > 0.01) return false; // ❌ never promote on “average quality” alone
  if (candidate.avgScore + 0.02 < baseline.avgScore) return false;
  return true;
}

# ✅ parse automatic eval output pointers (customize to actual result schema)
import json, boto3
s3 = boto3.client("s3")

def load_scores(bucket: str, prefix: str) -> dict:
    # list result objects; aggregate mean score + per-category breakdown
    agg = {"n": 0, "sum": 0.0, "by_category": {}}
    # ... read JSONL results written by the evaluation job ...
    return agg

❌ Evaluating only “happy path” LeetCode-style prompts: production agents fail on messy repos, partial files, and injection. Your dataset must look like production traces (Athena can mine hard cases).

Wire to Prompt Management aliases

Flow that actually ships:

1. Author candidate prompt in Prompt Management as draft
2. Run Bedrock Model Evaluation: prod alias vs draft
3. If gate passes, move alias prod → new version
4. Archive eval report under Object Lock prefix for audit

# ✅ conceptual — do not flip prod alias without eval artifact URI recorded
def promote_prompt(prompt_id: str, new_version: str, eval_s3_uri: str, gate_ok: bool):
    if not gate_ok:
        raise RuntimeError(f"eval failed: {eval_s3_uri}")
    # bedrock-agent update prompt alias → new_version
    # write audit row: who, when, eval_s3_uri, metrics
    return {"promoted": new_version, "eval": eval_s3_uri}

Prompt Flows remain for multi-step runtime graphs; Model Evaluation is the offline/pre-prod scorer. Do not confuse the two in runbooks.

Human evaluation when automatic metrics lie

Use human workflows in Bedrock Model Evaluation (or your own Labeling) when:

• Rubric quality matters more than BLEU-like overlap
• Safety / refusal grading needs expert eyes
• You are changing tool-calling style guides

Sample 5–10% of tasks; require dual review on security categories. Store human scores beside automatic ones in S3 for drift analysis.

Cost and governance

Eval jobs invoke models — often many times per task. Tag jobs workload=coding-agent, set Budgets, and prefer smaller models for draft iteration then confirm on the production model family. Deny broad bedrock:* in CI roles; allow only eval + invoke on listed model IDs.

Production checklist

• [ ] Golden dataset in S3; versioned (coding-vN); owners assigned
• [ ] Bedrock eval role least privilege; output bucket encrypted
• [ ] CI job: baseline vs candidate; fail pipeline on regression
• [ ] Prompt Management prod alias only moved by gated pipeline
• [ ] Safety / injection cases mandatory in every eval
• [ ] Results retained ≥ 90 days (Object Lock on audit packs)
• [ ] Budgets alarm on eval + Bedrock spend
• [ ] Quarterly human review calibration vs automatic judge

FAQ

Q: How is this different from offline RAG eval?
A: RAG eval focuses on retrieval hit rates and grounded answers. Model Evaluation here scores agent generation / prompt / model variants for coding tasks — including tool-oriented prompts. Use both if you have a Knowledge Base path.

Q: Can I eval tool-calling agents end-to-end?
A: Automatic jobs score model outputs against datasets. Full tool loops (sandbox + tests) still need your harness — use Bedrock eval for the LLM step, then run integration evals in CI against Fargate sandboxes.

Q: How often?
A: On every candidate prompt/model change that could reach prod, plus a weekly smoke eval on the frozen golden set to catch platform drift.

Related reading

• Amazon Bedrock Prompt Management
• Bedrock Prompt Flows
• Athena: Query Coding-Agent Eval Traces
• S3 Object Lock: Immutable Artifacts

Bedrock Model Evaluation turns prompt promotion from folklore into a scored gate: golden coding tasks, automatic metrics, optional human review, and Prompt Management aliases that only move when the numbers say so. Wire it into CI before the next “small prompt tweak” becomes a Monday outage.

Last updated on September 26, 2026