Synthetic Evals for Agents: Canary Coding Tasks Inside Every CI

Agent quality dies quietly — a prompt tweak, a model swap, a broken tool schema — and accept-rate dashboards stay green. The unfair advantage is fixed canary coding tasks in CI: tiny, deterministic jobs with golden diffs and tool traces that must pass on every agent-stack release before developers feel the regression.

⚡ TL;DR: Maintain 10–30 frozen tasks (bugfix, rename, test authoring) with golden outputs and allowed tool traces. Run on every prompt/model/tool change; fail the release on score drop. Not a substitute for blind A/B on real tickets — a smoke alarm. Pair with RAG Evaluation on AWS.

Canary task format

# evals/canaries/rename-helper.yaml
id: rename-helper-01
repo_fixture: fixtures/tiny-lib
prompt: |
  Rename export `add` to `addNumbers` and update callers.
  Do not change behavior.
budget:
  max_tool_calls: 12
  max_tokens: 8000
expect:
  diff_golden: goldens/rename-helper-01.diff
  tests: ["pnpm test"]
  forbidden_paths: ["pnpm-lock.yaml", ".github/**"]

// evals/run-canary.ts
export async function runCanary(c: Canary): Promise<Score> {
  const session = await agent.run({ prompt: c.prompt, cwd: c.repo_fixture });
  const diff = await gitDiff(c.repo_fixture);
  const match = normalizeDiff(diff) === read(c.expect.diff_golden);
  const toolsOk = session.toolCalls.length <= c.budget.max_tool_calls;
  const forbidden = touchedForbidden(diff, c.expect.forbidden_paths);
  // ✅ binary gates for smoke; keep graded rubrics elsewhere
  return { pass: match && toolsOk && !forbidden && testsGreen(c) };
}

Where canaries gate

# ✅ CI on agent-stack repo
pnpm evals:canaries --fail-under 0.95
# ❌ Shipping prompt changes with “tried a few chats locally”

Avoid canary overfit

Rotate 10% of tasks quarterly; keep a held-out set. Prefer tasks that assert behavior (tests green) plus a golden diff for mechanical edits. For flaky agent nondeterminism, allow N seeds with majority pass — never hide failures by raising temperature. Tie to Deterministic Replay so failing canaries dump replay bundles.

Closing checklist

✅ Dos
– ✅ Freeze fixtures + golden diffs in git
– ✅ Run on every prompt/model/tool change
– ✅ Fail release under threshold (e.g. 95%)
– ✅ Budget tool calls and forbid lockfile touches
– ✅ Dump replay artifacts on failure

❌ Don’ts
– ❌ Don’t use canaries as the only eval (still run human A/B)
– ❌ Don’t update goldens casually to “make CI green”
– ❌ Don’t allow network/prod tools in canary sandboxes
– ❌ Don’t measure only accept rate
– ❌ Don’t skip canaries for “emergency” model swaps

Related reading

• Evaluating AI Coding Tools: Blind A/B Tests on Real Tickets
• RAG Evaluation on AWS: Hit Rate, Faithfulness, and Cost Curves
• Deterministic Replay: Agent Sessions You Can Debug in Postmortems
• LLM Output Validators: Parse Trees Before Accepting Any Patches

Last updated on September 11, 2026