LLM Output Validators: Parse Trees Before Accepting Any Patches

An LLM that emits a patch which does not parse is not “almost right” — it is a failed tool call. Treat syntax, typecheck, and import resolution as hard validators in front of the working tree. Soft prompts (“please emit valid TypeScript”) are not a substitute for a gate that refuses to write bytes.

⚡ TL;DR: Parse every proposed file with the real language toolchain before git apply. Fail closed on parse errors, unresolved imports, and typecheck deltas outside an allowlist. Retry with the validator error as structured feedback at most N times, then escalate to a human. Aligns with Structured Outputs for Codegen and Cursor Agent Mode: Multi-File Refactors That Survive CI Gates.

Validator pipeline before any write

// validate-patch.ts
import ts from "typescript";
import { applyPatch } from "./diff";
import { execa } from "execa";

export type Validation = { ok: true } | { ok: false; stage: string; detail: string };

export async function validatePatch(opts: {
  repoRoot: string;
  unifiedDiff: string;
  maxTypeErrors?: number;
}): Promise<Validation> {
  const tree = applyPatch(opts.repoRoot, opts.unifiedDiff, { dryRun: true });
  if (!tree.ok) return { ok: false, stage: "apply", detail: tree.error };

  for (const file of tree.files) {
    if (!/\.(ts|tsx|js|jsx)$/.test(file.path)) continue;
    const sf = ts.createSourceFile(file.path, file.content, ts.ScriptTarget.Latest, true);
    const parseDiags = (sf as any).parseDiagnostics ?? [];
    // ✅ Real parse tree — not a regex "looks like code"
    if (parseDiags.length) {
      return {
        ok: false,
        stage: "parse",
        detail: parseDiags.map((d: ts.Diagnostic) => ts.flattenDiagnosticMessageText(d.messageText, "\n")).join("\n"),
      };
    }
  }

  // Materialize to a temp worktree for typecheck
  const { exitCode, stderr } = await execa(
    "pnpm",
    ["exec", "tsc", "-p", "tsconfig.json", "--pretty", "false", "--noEmit"],
    { cwd: tree.tempRoot, reject: false },
  );
  if (exitCode !== 0) {
    const errors = stderr.split("\n").filter((l) => /error TS/.test(l));
    if (errors.length > (opts.maxTypeErrors ?? 0)) {
      return { ok: false, stage: "typecheck", detail: errors.slice(0, 40).join("\n") };
    }
  }
  return { ok: true };
}

❌ Writing the patch first and “fixing CI later.”
✅ Dry-run apply → parse → typecheck → only then write.

Structured retry with validator feedback

async function generateUntilValid(agent: Agent, prompt: string, budget = 3) {
  let feedback = "";
  for (let i = 0; i < budget; i++) {
    const diff = await agent.proposePatch(prompt + feedback);
    const v = await validatePatch({ repoRoot: process.cwd(), unifiedDiff: diff });
    if (v.ok) return diff;
    // ✅ Feed machine-readable failure back — not "try again"
    feedback = `\n\nPREVIOUS_PATCH_REJECTED stage=${v.stage}\n${v.detail}\nEmit a corrected unified diff only.`;
  }
  throw new Error("validator_budget_exhausted");
}

Same discipline as JSON schema gates in Structured Outputs for Codegen. Prefer AST-guided transforms when mechanical — see companion topics on AST intents when you publish them; until then keep validators ruthless.

Import resolution and forbidden paths

const FORBIDDEN = [/^pnpm-lock\.yaml$/, /\/generated\//, /^\.env/];

function assertImportsResolve(file: string, content: string, resolver: Resolver) {
  for (const spec of extractImportSpecifiers(content)) {
    const resolved = resolver.resolve(file, spec);
    if (!resolved) throw new Error(`unresolved_import:${spec}`);
  }
}

function assertNotForbidden(paths: string[]) {
  for (const p of paths) {
    if (FORBIDDEN.some((re) => re.test(p))) throw new Error(`forbidden_path:${p}`);
  }
}

Hook these into agent sandboxes as in Claude Code Hooks: Gate Risky Shell Commands Before CI Runs.

What “valid” means per language

Infra-touched diffs still need human review and the same CI gates as Cursor Rules for AWS CDK; require terraform validate locally before any apply.

Closing checklist

• [ ] No patch enters the working tree without parse success
• [ ] Typecheck / compile runs in a temp worktree; baseline errors allowlisted
• [ ] Unresolved imports and forbidden paths fail closed
• [ ] Retry loop caps at N; validator stderr becomes next prompt context
• [ ] Metrics: reject rate by stage (parse / typecheck / import)
• [ ] Semantic review still runs after validators — see Semantic Diff Review

Related reading

• Structured Outputs for Codegen: JSON Schemas That Actually Compile
• Cursor Agent Mode: Multi-File Refactors That Survive CI Gates
• Semantic Diff Review: AI That Flags Behavioral Drift Only
• Claude Code Hooks: Gate Risky Shell Commands Before CI Runs

Last updated on September 11, 2026