Semantic Diff Review: AI That Flags Behavioral Drift Only

Most AI PR comments are noise: “consider renaming this variable,” “add a blank line,” “imports aren’t sorted.” Seniors mute the bot. Semantic diff review flips the objective — ignore formatting-only churn and spend the context window on AST-level and behavioral drift: contract breaks, authz regressions, and performance cliffs.

⚡ TL;DR: Preprocess the PR into a semantic diff (AST-aware, ignore whitespace/import reorder). Feed the reviewer changed symbols + callers + contract tests only. Prompt for severity-tagged findings with reproducible evidence. Drop nits below a threshold. Illustrative win: 70% fewer comments, higher accept rate on high-severity findings.

Build a semantic diff, not a unified diff dump

# Prefer structural diffs when available
git diff --ignore-all-space origin/main...HEAD > /tmp/raw.diff
# Illustrative: emit changed symbols via scip / lsif / ts-morph
node tools/semantic-diff.js --base origin/main --out /tmp/semantic.json

// semantic-diff.js excerpt — focus on behavioral surface
import { Project } from "ts-morph";

export function changedExports(baseFiles: string[], headFiles: Map<string, string>) {
  const findings: { path: string; name: string; kind: string }[] = [];
  const project = new Project({ skipAddingFilesFromTsConfig: true });
  for (const [path, text] of headFiles) {
    const sf = project.createSourceFile(path, text, { overwrite: true });
    for (const [name, decls] of sf.getExportedDeclarations()) {
      findings.push({ path, name, kind: decls[0].getKindName() });
    }
  }
  return findings;
}

✅ Pass: changed function bodies, signature diffs, OpenAPI deltas, IAM policy diffs.
❌ Pass: prettier-only files, import resort, generated lockfile blobs.

Context packing for the reviewer model

Follow the diff-scoped pattern: changed symbols + one-hop callers + existing tests.

{
  "pr": 1842,
  "behavioral_diff": [
    {
      "symbol": "InvoiceService.create",
      "signature_before": "(input: CreateInvoice) => Promise<Invoice>",
      "signature_after": "(input: CreateInvoice, opts?: {skipTax?: boolean}) => Promise<Invoice>",
      "callers": ["CheckoutController.complete"],
      "tests": ["invoice.service.test.ts"]
    }
  ],
  "contracts": ["openapi.yaml#/paths/~1invoices/post"],
  "forbidden_nits": ["style", "naming-bike-shed", "import-order"]
}

System prompt excerpt:

You are a senior code reviewer. Only report:
1) contract/API breaks 2) authz/authn regressions 3) data-loss or idempotency bugs
4) obvious p99 performance cliffs 5) secret leakage.
Ignore formatting, naming taste, and optional refactors.
Severity: blocker|major|minor. Require file:line evidence.
If no behavioral risk, reply NO_FINDINGS.

Tie into AI Code Review Bots IAM so the bot can read the PR but not the entire org.

Auth and contract checks that beat vibes

# fail the review job if OpenAPI response schema drifted without changelog
import json, sys
before, after = json.load(open(sys.argv[1])), json.load(open(sys.argv[2]))
# illustrative: compare /invoices post 200 schema hash
assert before["paths"]["/invoices"]["post"]["responses"]["200"] == \
       after["paths"]["/invoices"]["post"]["responses"]["200"], "schema drift — require dual-run note"

For authz, require the model to cite the middleware or guard that still runs after the change — or flag removal.

Suppress nits ruthlessly

# reviewer-config.yml
max_comments: 8
drop_severities: [nit]
block_on: [blocker, major]
ignore_paths:
  - "**/*.generated.*"
  - "**/pnpm-lock.yaml"
prompt_version: semantic-review@2026-09-11

Track metrics: comments/PR, human thumbs-up rate, escaped sev-1s. If thumbs-up falls, tighten the prompt — don’t add more verbosity. Align commit hygiene with Agentic Git Workflows so reviewers see atomic intents.

Wire into CI without blocking every PR forever

Run semantic review as a non-blocking check that posts comments, and only fail the build on blocker findings the team has calibrated for two weeks.

# .github/workflows/semantic-review.yml — illustrative
name: semantic-review
on: pull_request
jobs:
  review:
    runs-on: ubuntu-latest
    permissions:
      contents: read
      pull-requests: write
    steps:
      - uses: actions/checkout@v4
        with: { fetch-depth: 0 }
      - run: node tools/semantic-diff.js --base origin/${{ github.base_ref }} --out semantic.json
      - run: node tools/run-reviewer.js --in semantic.json --prompt-version semantic-review@2026-09-11
        env:
          BEDROCK_MODEL: ${{ vars.REVIEW_MODEL }}
          BUDGET_PR_TOKENS: "200000"

Cap tokens per PR (LLM cost controls). If the reviewer times out, fail open with a warning — never wedge merges on vendor latency alone unless you have a hard compliance gate.

Closing checklist

✅ Dos
– ✅ Precompute semantic/AST diffs before prompting
– ✅ Include callers and contract artifacts in context
– ✅ Severity-tag findings; auto-drop nits
– ✅ Version the reviewer prompt; A/B on real PRs
– ✅ Measure human accept rate, not comment volume

❌ Don’ts
– ❌ Don’t paste the entire unified diff into the model
– ❌ Don’t allow style bikesheds to page authors
– ❌ Don’t grant the bot monorepo admin tokens
– ❌ Don’t treat “LGTM from AI” as a required check alone
– ❌ Don’t review generated files without an allowlist exception

Related reading

• AI Code Review Bots: IAM, Secrets, and Least-Privilege Pipelines
• Cursor Rules for TypeScript Monorepos: Make AI Edits Stick
• Agentic Git Workflows: Atomic Commits From Noisy LLM Diffs (companion)
• LLM Coding Agents on AWS: Safe Tool Sandboxes with Lambda

Last updated on September 11, 2026