Day 8: Structured Outputs That Compile

Freeform prose is fine for chat. It is a liability for agents that must call APIs, open PRs, or write IAM. Day 8 is about structured outputs that compile: schemas, constrained decoding / tool-choice, and post-validators so illegal fields never reach side effects.

⚡ TL;DR: Define JSON Schema (or equivalent) for every tool argument and every agent plan object. Prefer provider tool-calling or constrained decoding. Validate with a strict parser; reject unknown properties; retry once with the error — then fail closed.

Schema first, prompt second

{
  "type": "object",
  "additionalProperties": false,
  "required": ["action", "path", "patch"],
  "properties": {
    "action": {"enum": ["apply_patch", "run_tests", "comment"]},
    "path": {"type": "string", "minLength": 1, "maxLength": 512},
    "patch": {"type": "string", "maxLength": 20000},
    "idempotency_key": {"type": "string", "pattern": "^[a-zA-Z0-9_-]{8,64}$"}
  }
}

❌ “Return JSON” in prose with no schema and JSON.parse optimism. Models invent filePath vs path, nest surprises, and drop required keys under pressure.

Keep schemas in the same repo as handlers and fail CI when they drift (OpenAPI generate + typecheck).

Tool-choice and constrained decoding

When the platform supports it (Bedrock toolConfig, OpenAI tools, etc.):

• Register tools with schemas.
• Set tool choice to required or a specific tool when the turn must be a side effect.
• Keep additionalProperties: false wherever the API allows.

# ✅ Validate before side effects
from jsonschema import Draft202012Validator

validator = Draft202012Validator(APPLY_PATCH_SCHEMA)

def handle_tool(raw: dict):
    errors = sorted(validator.iter_errors(raw), key=lambda e: e.path)
    if errors:
        raise ValidationError([e.message for e in errors])
    return apply_patch(**raw)

If the model returns prose anyway, do not “best effort” extract with regex for privileged actions. Ask for a structured retry once; then fail.

Validators beyond JSON Schema

Schema correctness ≠ semantic safety:

• Path allowlists (src/** only).
• Patch size and file count caps.
• Ban shell metacharacters in arguments you will interpolate (better: never interpolate — pass argv arrays).
• Idempotency keys for mutating tools (Day 11).

// ✅ Semantic gate after schema pass
function assertSafePatch(args: ApplyPatch) {
  if (args.path.includes("..") || args.path.startsWith("/etc")) {
    throw new Error("path_not_allowed");
  }
  if (args.patch.length > 20_000) throw new Error("patch_too_large");
}

UX for structured modes

Streaming structured JSON is awkward. Prefer:

• Stream status events (“planning”, “calling tool X”) while buffering the tool args, or
• Use partial JSON parsers carefully only for non-side-effect previews.

Never apply a half-validated mutating tool call.

Closing checklist

• [ ] JSON Schema (strict) for every tool and plan object
• [ ] additionalProperties: false and enums for actions
• [ ] Provider tool-calling / constrained decoding where available
• [ ] Server-side validation before any side effect
• [ ] One structured retry, then fail closed
• [ ] CI check: schema ↔ handler types in sync

Worked example: PR bot args

Require {action, path, patch, idempotency_key} with additionalProperties:false. When the model emits file instead of path, validators return a structured error and the agent retries once. The mutating apply_patch tool never sees illegal shapes.

Log schema failures as a first-class metric — spikes mean prompt drift or model change.

Failure modes to watch

• Regex JSON extraction for privileged tools.
• Schemas in prompts only, not enforced server-side.
• Streaming apply of partial objects.
• Enum-less freeform actions (do_whatever).

Field notes from production

When providers add constrained decoding, keep your server-side validator anyway — defense in depth and portability across models. Version schemas (application/vnd.acme.patch.v2+json) so old agents fail loudly instead of partially applying v1 fields.

Implementation sketch

# Implementation sketch: reject unknown fields
def parse_tool(raw: dict, schema) -> dict:
    Draft202012Validator(schema).validate(raw)
    return raw

Operator addendum

Property-test your validators: generate random JSON and assert handlers never execute on invalid shapes. This catches the ‘we only validated in the happy path’ bug that appears during incident hotfixes.

End-to-end compile pipeline

Think of structured output like a compiler frontend: parse → typecheck (schema) → semantic checks → emit side effects. Each stage returns typed errors the agent can consume. Add golden fixtures of model outputs you have seen in the wild (wrong keys, nulls, trailing prose) and assert the pipeline rejects them. For TypeScript handlers, generate types from JSON Schema in CI so drift is a build break, not an on-call surprise. Prefer tool_choice=required when a turn must be a function call; falling back to prose mid-mutation is how half-applied patches happen.

Extended discussion

Return to the core angle for Day 8: JSON Schema, tool-choice, and validators so the model cannot invent fields. That sentence is the acceptance lens for every design review this week. If a proposed change does not make this angle easier to measure or enforce, it is a distraction.

Write down three metrics you will look at after shipping Day 8 ideas, schedule a 45-minute readout, and archive the notes next to the eval artifacts. Architecture without a readout becomes slideshow archaeology.

Pair this day with the adjacent lessons in the series navigation below. Forward links exist so you can keep momentum; backward links exist so you can repair foundations when a later lab fails for boring earlier reasons.

Practically, allocate half a day to implement the smallest vertical slice, half a day to wire measurement, and refuse to polish UI until both are done. This ordering is how bootcamp projects stay honest under time pressure.

Revisit assumptions whenever the model ID, embedding ID, or index alias changes — treat those as breaking changes for Day 8 behaviors, with the same seriousness as a database migration. Canary first, then promote.

Series navigation

← Day 7 · Day 9 →

Last updated September 11, 2026