Day 20: Project: PR Review Bot With Diff-Scoped Context

Day 20 is a project lab: ship a PR review bot in the spirit of Cursor/Claude-style reviewers that only sees the diff and a small set of CODEOWNERS-selected context files. Unlimited repo RAG feels powerful and produces confident wrong comments. Scoped context produces fewer, sharper findings.

⚡ TL;DR: Fetch the pull diff + changed file contents. Resolve CODEOWNERS for those paths and attach only listed high-signal files (or summaries). Run structured review output. Post comments idempotently. Eval on golden PRs. Refuse to review when the diff exceeds budget.

Project lab — architecture

GitHub webhook (PR opened/sync)
  → verify signature
  → load diff + patch (token-budgeted)
  → CODEOWNERS resolve → context files
  → Bedrock structured review (Day 8)
  → idempotent review comments
  → metrics + eval hooks

Project lab — build steps

1. Webhook: PR opened / synchronize; verify secret; enqueue SQS (do not review inline if slow).
2. Diff pack: unified diff truncated to token budget; prefer per-file caps.
3. Context pack: for each changed path, include the file (or AST chunk) + CODEOWNERS-required companions (e.g. auth middleware when handlers change).
4. System contract: only comment on bugs, security, and missing tests; no style bikesheds unless configured.
5. Structured output: {findings:[{path,line,severity,rationale,evidence}]} validated.
6. Post comments: idempotency key per pr+sha+finding_hash to avoid spam on re-runs.
7. Evals: 10 golden PRs with expected finding categories (Day 9).
8. Demo: open a PR with a deliberate authz bug; show the bot cites the diff hunk.

# ✅ Scope enforcement
def build_pack(pr):
    diff = pr.unified_diff()
    paths = pr.changed_paths()
    owners_files = codeowners.context_for(paths)
    pack = pack_tokens(diff=diff, files=owners_files, limit=20_000)
    assert set(pack.files) <= set(owners_files) | set(paths)
    return pack

❌ Embedding the entire monorepo “just in case” — classic lost-in-the-middle plus cost explosion (Days 1, 15).

What “done” looks like

• Comments only reference lines in the diff or explicitly attached files.
• Re-sync does not duplicate comments (idempotency).
• Goldens gate prompt changes.
• Budget refuse when diff too large: ask humans to split the PR.
• Logs redacted (Day 18); no raw secrets from .env diffs — skip or redact those files.

Field notes from production

Noise kills adoption. Start with severity>=high only. Measure “comments addressed vs dismissed.” If dismiss rate > 50%, your prompt is a scold, not a reviewer. Pair with CI linters for mechanical issues so the model focuses on semantic bugs.

Implementation sketch

FINDING_SCHEMA = {
  "type": "object",
  "additionalProperties": False,
  "required": ["findings"],
  "properties": {
    "findings": {"type": "array", "items": {
      "type": "object",
      "required": ["path", "line", "severity", "rationale"],
      "additionalProperties": False,
      "properties": {
        "path": {"type": "string"},
        "line": {"type": "integer"},
        "severity": {"enum": ["low","medium","high"]},
        "rationale": {"type": "string"},
        "evidence": {"type": "string"}
      }
    }}
  }
}

Closing checklist

• [ ] Diff-scoped pack with hard token limit
• [ ] CODEOWNERS-driven extra context only
• [ ] Structured findings validated before post
• [ ] Idempotent comments per sha
• [ ] Golden PR eval in CI
• [ ] Redact secret-bearing files from context

Project lab hardening path

After the thin slice works, add: severity thresholds, path allowlists, secret file skips, rate limits per repo, and a “bot ignored” reaction metric. Consider posting a single summary comment plus inline findings to reduce noise. Document how to disable the bot per repo via CODEOWNERS or label. Ship the disable switch on day one — forced review bots without escape hatches get forked out of org workflows.

Extended discussion

Return to the core angle for Day 20: Cursor/Claude-style reviewer that only sees the diff plus CODEOWNERS files. That sentence is the acceptance lens for every design review this week. If a proposed change does not make this angle easier to measure or enforce, it is a distraction.

Write down three metrics you will look at after shipping Day 20 ideas, schedule a 45-minute readout, and archive the notes next to the eval artifacts. Architecture without a readout becomes slideshow archaeology.

Pair this day with the adjacent lessons in the series navigation below. Forward links exist so you can keep momentum; backward links exist so you can repair foundations when a later lab fails for boring earlier reasons.

Practically, allocate half a day to implement the smallest vertical slice, half a day to wire measurement, and refuse to polish UI until both are done. This ordering is how bootcamp projects stay honest under time pressure.

Revisit assumptions whenever the model ID, embedding ID, or index alias changes — treat those as breaking changes for Day 20 behaviors, with the same seriousness as a database migration. Canary first, then promote.

Revisit assumptions whenever the model ID, embedding ID, or index alias changes — treat those as breaking changes for Day 20 behaviors, with the same seriousness as a database migration. Canary first, then promote.

Revisit assumptions whenever the model ID, embedding ID, or index alias changes — treat those as breaking changes for Day 20 behaviors, with the same seriousness as a database migration. Canary first, then promote.

Revisit assumptions whenever the model ID, embedding ID, or index alias changes — treat those as breaking changes for Day 20 behaviors, with the same seriousness as a database migration. Canary first, then promote.

Revisit assumptions whenever the model ID, embedding ID, or index alias changes — treat those as breaking changes for Day 20 behaviors, with the same seriousness as a database migration. Canary first, then promote.

Series navigation

← Day 19 · Day 21 →

Last updated September 11, 2026