Semantic Diff Review: AI That Flags Behavioral Drift Only

Semantic Diff Review: AI That Flags Behavioral Drift Only
18 views

This guide covers semantic diff review with production-focused examples and trade-offs.

Most AI PR comments are noise: “consider renaming this variable,” “add a blank line,” “imports aren’t sorted.” Seniors mute the bot. Semantic diff review flips the objective — ignore formatting-only churn and spend the context window on AST-level and behavioral drift: contract breaks, authz regressions, and performance cliffs.
⚡ TL;DR: Preprocess the PR into a semantic diff (AST-aware, ignore whitespace/import reorder). Feed the reviewer changed symbols + callers + contract tests only. Prompt for severity-tagged findings with reproducible evidence. Drop nits below a threshold. Illustrative win: 70% fewer comments, higher accept rate on high-severity findings.

Build a semantic diff, not a unified diff dump

bash
# Prefer structural diffs when available
git diff --ignore-all-space origin/main...HEAD > /tmp/raw.diff
# Illustrative: emit changed symbols via scip / lsif / ts-morph
node tools/semantic-diff.js --base origin/main --out /tmp/semantic.json
typescript
// semantic-diff.js excerpt — focus on behavioral surface
import { Project } from "ts-morph";

export function changedExports(baseFiles: string[], headFiles: Map<string, string>) {
  const findings: { path: string; name: string; kind: string }[] = [];
  const project = new Project({ skipAddingFilesFromTsConfig: true });
  for (const [path, text] of headFiles) {
    const sf = project.createSourceFile(path, text, { overwrite: true });
    for (const [name, decls] of sf.getExportedDeclarations()) {
      findings.push({ path, name, kind: decls[0].getKindName() });
    }
  }
  return findings;
}
✅ Pass: changed function bodies, signature diffs, OpenAPI deltas, IAM policy diffs.
❌ Pass: prettier-only files, import resort, generated lockfile blobs.

Context packing for the reviewer model

Follow the diff-scoped pattern: changed symbols + one-hop callers + existing tests.
json
{
  "pr": 1842,
  "behavioral_diff": [
    {
      "symbol": "InvoiceService.create",
      "signature_before": "(input: CreateInvoice) => Promise<Invoice>",
      "signature_after": "(input: CreateInvoice, opts?: {skipTax?: boolean}) => Promise<Invoice>",
      "callers": ["CheckoutController.complete"],
      "tests": ["invoice.service.test.ts"]
    }
  ],
  "contracts": ["openapi.yaml#/paths/~1invoices/post"],
  "forbidden_nits": ["style", "naming-bike-shed", "import-order"]
}
System prompt excerpt:
text
You are a senior code reviewer. Only report:
1) contract/API breaks 2) authz/authn regressions 3) data-loss or idempotency bugs
4) obvious p99 performance cliffs 5) secret leakage.
Ignore formatting, naming taste, and optional refactors.
Severity: blocker|major|minor. Require file:line evidence.
If no behavioral risk, reply NO_FINDINGS.
Tie into AI Code Review Bots IAM so the bot can read the PR but not the entire org.

Auth and contract checks that beat vibes

python
# fail the review job if OpenAPI response schema drifted without changelog
import json, sys
before, after = json.load(open(sys.argv[1])), json.load(open(sys.argv[2]))
# illustrative: compare /invoices post 200 schema hash
assert before["paths"]["/invoices"]["post"]["responses"]["200"] == \
       after["paths"]["/invoices"]["post"]["responses"]["200"], "schema drift — require dual-run note"
For authz, require the model to cite the middleware or guard that still runs after the change — or flag removal.

Suppress nits ruthlessly

yaml
# reviewer-config.yml
max_comments: 8
drop_severities: [nit]
block_on: [blocker, major]
ignore_paths:
  - "**/*.generated.*"
  - "**/pnpm-lock.yaml"
prompt_version: semantic-review@2026-09-11
Track metrics: comments/PR, human thumbs-up rate, escaped sev-1s. If thumbs-up falls, tighten the prompt — don’t add more verbosity. Align commit hygiene with Agentic Git Workflows so reviewers see atomic intents.

Wire into CI without blocking every PR forever

Run semantic review as a non-blocking check that posts comments, and only fail the build on blocker findings the team has calibrated for two weeks.
yaml
# .github/workflows/semantic-review.yml — illustrative
name: semantic-review
on: pull_request
jobs:
  review:
    runs-on: ubuntu-latest
    permissions:
      contents: read
      pull-requests: write
    steps:
      - uses: actions/checkout@v4
        with: { fetch-depth: 0 }
      - run: node tools/semantic-diff.js --base origin/${{ github.base_ref }} --out semantic.json
      - run: node tools/run-reviewer.js --in semantic.json --prompt-version semantic-review@2026-09-11
        env:
          BEDROCK_MODEL: ${{ vars.REVIEW_MODEL }}
          BUDGET_PR_TOKENS: "200000"
Cap tokens per PR (LLM cost controls). If the reviewer times out, fail open with a warning — never wedge merges on vendor latency alone unless you have a hard compliance gate.

Closing checklist

✅ Dos – ✅ Precompute semantic/AST diffs before prompting – ✅ Include callers and contract artifacts in context – ✅ Severity-tag findings; auto-drop nits – ✅ Version the reviewer prompt; A/B on real PRs – ✅ Measure human accept rate, not comment volume ❌ Don’ts – ❌ Don’t paste the entire unified diff into the model – ❌ Don’t allow style bikesheds to page authors – ❌ Don’t grant the bot monorepo admin tokens – ❌ Don’t treat “LGTM from AI” as a required check alone – ❌ Don’t review generated files without an allowlist exception

Related reading

Last updated on September 11, 2026

Related: Spec-First AI Development: OpenAPI Remains the Only Source of Truth · AI Code Review Automation: Build a GitHub Bot That Reviews PRs Like a Senior Engineer · Day 93: SLOs for AI Features

Related: Prompt Engineering for Developers: The Patterns That 10x Your AI Coding Output · Vibe Coding: How Senior Developers Actually Use AI to Write Code Faster · Day 80: Project: Platform AI Gateway

Watch: 100 Days of AI on YouTube

Short videos from CheatCoders, one AI engineering topic per day.

Open the playlist on YouTube · CheatCoders on YouTube

Deep-dive PDF

Get the expanded guide for this post — extra diagrams-style checklists, failure modes, and a production walkthrough. Free when you subscribe to CheatCoders.

Already subscribed? or open the subscribe page.


Discover more from CheatCoders

Subscribe to get the latest posts sent to your email.

Comments

No comments yet. Why don’t you start the discussion?

Leave a comment

No account needed. Name and email are optional.