AI Pair Programming Metrics: Cycle Time Without Vanity Accept Scores

“80% accept rate” is a vanity metric. Engineers accept junk to dismiss the popup, then rewrite it. Measure what shipping cares about: cycle time, human edit distance after accept, revert rate, and review latency — not how often someone hit Tab.

⚡ TL;DR: Instrument suggestion → accept → final-landed diff. Track normalized edit distance, time-to-merge, review comment severity, and revert rate by tool. Kill dashboards that only show accept %. Pair with Evaluating AI Coding Tools and Semantic Diff Review.

Metrics that survive skepticism

// metrics/aiPair.ts
export type SuggestionEvent = {
  id: string;
  tool: "cursor" | "copilot" | "claude";
  acceptedText: string;
  file: string;
  ts: number;
};

export function netAssist(accepted: string, finalHunk: string): number {
  const d = levenshtein(accepted, finalHunk);
  const denom = Math.max(accepted.length, 1);
  return Math.max(0, 1 - d / denom);
}

Instrumentation without creepy keylogging

Collect events from editor telemetry SDKs and CI — not keystroke loggers. Hash file paths if needed; never ship file contents to a third-party analytics SaaS without DPA review.

{
  "event": "ai_suggestion_resolved",
  "tool": "cursor",
  "net_assist": 0.72,
  "pr": 1842,
  "cycle_hours": 6.5,
  "reverted_7d": false
}

Experiment design

Blind A/B by team or repo as in Evaluating AI Coding Tools. Hold ticket difficulty roughly constant (label size:S/M/L). Primary outcome: cycle time; secondary: revert rate. Accept rate is exploratory only.

-- warehouse sketch
select tool,
       approx_percentile(cycle_hours, 0.5) as p50_cycle,
       avg(net_assist) as avg_net_assist,
       avg(reverted_7d::int) as revert_rate
from ai_pair_facts
where week >= current_date - interval '28 days'
group by 1;

Anti-gaming

❌ Leaderboards for “most accepts.”
✅ Celebrate drop in revert rate and review blockers. Cap or ignore accepts with net_assist < 0.2 in rollups so Tab-spam does not inflate success.

Dashboard layout seniors actually use

Ship three panels, nothing more in v1:

1. Outcomes — p50/p90 cycle hours by tool vs control cohort.
2. Stickiness — distribution of net_assist (0–0.2 / 0.2–0.6 / 0.6–1.0).
3. Quality tax — revert rate and blocker-comment rate.

Anything else (keystrokes, ghost-text render counts) stays in a buried debug view. Product marketing gets outcomes, not Tab metrics.

// ci/annotate-pr-with-ai-stats.ts — optional PR comment
export function formatAiStats(s: { netAssist: number; tool: string }) {
  if (s.netAssist < 0.2) {
    return `AI assist (${s.tool}): low stickiness — treat as draft only.`;
  }
  return `AI assist (${s.tool}): net_assist=${s.netAssist.toFixed(2)}`;
}

Closing checklist

✅ Dos
– ✅ Track net assist, cycle time, reverts, review severity
– ✅ A/B with real tickets and size labels
– ✅ Privacy-review telemetry payloads
– ✅ Segment by repo risk (payments vs docs)
– ✅ Revisit tool licenses against outcome deltas

❌ Don’ts
– ❌ Don’t optimize for raw accept rate
– ❌ Don’t publicize per-engineer vanity ranks
– ❌ Don’t ship source text to analytics without controls
– ❌ Don’t declare victory from a one-week pilot

Related reading

• Evaluating AI Coding Tools: Blind A/B Tests on Real Tickets
• Semantic Diff Review: AI That Flags Behavioral Drift Only
• AI PR Reviewers: Diff-Scoped Context Windows That Catch Real Bugs
• LLM Cost Controls: Token Budgets Per PR and Per Engineer

Last updated on September 11, 2026