AI Engineer Roadmap All lessons Interactive Quiz GitHub

Structured Outputs

Intermediate · 7 min read · Updated · By Saurabh Pathak

In short: Structured outputs make an LLM return data your code can parse, such as JSON that matches a schema, and you still validate every reply, because even a well-formed answer can be wrong.

Play the interactive lesson: Structured outputsRead on GitHub
On this page
  1. Explain it like I'm new
  2. Watch it
  3. How it works
  4. Code example
  5. When to use it / when not to
  6. Common mistakes
  7. Recap
  8. Interview questions
  9. Go deeper

Level: Intermediate · Reading time: 7 min · Phase: 3 · LLM Concepts

Before you start: Calling an LLM API and Hallucinations.

Words to know:

Explain it like I'm new

Imagine a paper form with boxes: Name, Category (tick one), Priority (1 to 3). A clerk checks it at the desk. If a box is empty or the tick is in the wrong place, the clerk hands it back and says what is wrong.

Free text is like a letter. A human can read it, but your program cannot reliably find the "category" in it. A model might write "Sure! Here is the JSON:" before the data, or call the field type one day and category the next. Structured outputs give you the form, and the validate-and-repair loop is the clerk.

Watch it

Structured outputs: Make the model return JSON your code can use, and still check it.
Structured outputs: Make the model return JSON your code can use, and still check it.

It plays on a loop. Read the caption in the box, one step at a time. Want to go at your own pace? Step through it with Previous and Next.

How it works

  1. Define a schema. Say which fields exist, their types, which are required, and which values are allowed.
  2. Ask for it. There are three common ways, from weakest to strongest:
    • Prompt only: "Reply with JSON like this." Cheap, but the model can drift.
    • JSON mode: the provider makes sure the output is valid JSON. It does not promise your field names.
    • Schema-constrained output: you pass a schema, and the provider restricts decoding so the output fits it.
    • Tool-call arguments: you describe a "function" with a schema, and the model fills in the arguments. This is the same mechanism agents use. See Tool use in Phase 4.

The exact parameter names and supported schema features differ by provider and change over time. Check the current provider docs.

  1. Extract. Pull the JSON object out of the reply. If the reply has prose or code fences around it, find the object inside.
  2. Validate against your schema. Collect a list of errors, not just the first one.
  3. Repair. If there are errors, send them back to the model: "Your last answer was invalid: ... Return only a corrected JSON object." Stop after a small number of tries.
  4. Give up cleanly. After the last try, return an error you can log, not a half-valid object.
flowchart LR
    A[Prompt + schema] --> B[LLM reply]
    B --> C[Extract JSON]
    C --> D{Valid?}
    D -->|yes| E[Use the data]
    D -->|no| F[Add errors to prompt]
    F -->|tries left| B
    F -->|no tries left| G[Fail with the errors]

Schema design tips.

What validation cannot do. It checks the shape, not the truth. {"category": "billing", "priority": 1, "tags": ["refund"]} is valid even if the customer was really reporting a bug. Check facts another way: compare to a source, add an evaluation set, or have a human review the risky cases. See Hallucinations for why a confident answer can still be wrong.

Worked example

Ask for a support ticket with category in billing, bug, other and priority 1 to 3:

  1. Try 1 returns category: "refund", priority: 7 and a wrong tag type. Validation lists all 3 errors.
  2. You send those errors back and ask for a corrected object.
  3. Try 3 passes every check, so the result {'category': 'bug', 'priority': 2, 'tags': ['login']} is used. This matches the demo's repaired result after 3 tries.

Code example

This demo is stdlib only. It has a tiny validator for a subset of JSON Schema (type, required, enum, minimum, maximum, nested objects, array items), not the whole spec. Run it: python 03-llm-concepts/code/llm_structured.py

from llm_structured import TICKET_SCHEMA, ScriptedLLM, extract_json, validate, repair_loop

prose = 'Sure! Here you go: {"category": "bug", "priority": 2, "tags": ["login"]} Hope it helps.'
print(extract_json(prose))   # -> {'category': 'bug', 'priority': 2, 'tags': ['login']}

bad = {"category": "refund", "priority": 7, "tags": ["ok", 5]}
print(validate(bad, TICKET_SCHEMA))
# -> ["$.category: 'refund' not in ['billing', 'bug', 'other']", '$.priority: 7 is above maximum 3', '$.tags[1]: expected string, got int']

llm = ScriptedLLM([
    "I think it is a billing issue, priority high.",
    '{"category": "billing", "priority": 9, "tags": []}',
    '{"category": "billing", "priority": 2, "tags": ["invoice"]}',
])
data, tries, errs = repair_loop(llm, "Classify: I was charged twice.", TICKET_SCHEMA, max_tries=3)
print(data, tries, errs)   # -> {'category': 'billing', 'priority': 2, 'tags': ['invoice']} 3 []

The ScriptedLLM plays back fixed replies: first prose with no JSON, then JSON with a bad priority, then a good answer. The full demo prints:

extracted: {'category': 'bug', 'priority': 2, 'tags': ['login']}
error: $.category: 'refund' not in ['billing', 'bug', 'other']
error: $.priority: 7 is above maximum 3
error: $.tags[1]: expected string, got int
result: {'category': 'billing', 'priority': 2, 'tags': ['invoice']} | tries: 3 | errors: []
second prompt ended with: Return only a corrected JSON object.
third prompt had: - $.priority: 9 is above maximum 3

The error messages are the useful part. The model sees exactly what to fix, and each message carries a path such as $.tags[1]. The tests in test_llm_structured.py check that errors are fed back into the next prompt, that the loop gives up after max_tries, that True is not accepted as an integer, and that a schema-valid but wrong answer passes (shape is not truth).

In real projects you would use a full JSON Schema validator or a data-model library, plus your provider's structured-output feature. Names and parameters change, so check the docs: JSON Schema and the Anthropic tool use guide or the OpenAI function calling guide.

When to use it / when not to

SituationAdvice
Your code reads the answer (classify, extract fields, route a ticket)Use structured outputs
The model must call a function or toolUse tool-call arguments, and validate them
A person reads the answer, and wording mattersPlain text is fine
The provider offers schema-constrained decodingUse it, but still validate
Only prompt-based JSON is availableUse validate-then-repair, with a retry limit
You need the answer to be trueValidation is not enough. Add retrieval, checks or review

Common mistakes

Recap

If you remember only three things:

Interview questions

<details> <summary><b>1. What is the difference between JSON mode and schema-constrained output?</b></summary>

JSON mode only guarantees that the output is valid JSON. It does not promise your fields, types or allowed values. Schema-constrained output limits generation to your schema, so the shape matches. Details vary by provider, so check the docs. In both cases, validate in your own code. </details>

<details> <summary><b>2. The model returns valid JSON that passes validation but the content is wrong. Why, and what now?</b></summary>

Validation checks shape, not truth. The model can fill a valid field with a wrong value. Add checks against a trusted source, an evaluation set to measure accuracy, and human review for high-risk cases. </details>

<details> <summary><b>3. Why feed the validation errors back instead of just retrying?</b></summary>

A plain retry often repeats the same mistake. The error list, such as "priority 9 is above maximum 3", tells the model exactly what to fix. Keep the number of tries small, and fail cleanly after that. </details>

Go deeper

Prev: Calling an LLM API · Next: Quantization · Back to phase: Phase 3