Structured Outputs from LLM APIs: JSON Schemas, Validation, and Recovery Patterns

Overview

LLMs generate text, while applications usually need dependable data. A support workflow may need a ticket category and priority. A document pipeline may need extracted entities, dates, and confidence notes. An AI workflow may need a list of actions that another service can execute. In each case, “please return JSON” is only a formatting request, not a complete reliability strategy.

A production-ready structured-output workflow separates four concerns:

1. Generation: instruct the model to produce the required fields and values.
2. Conformance: use a JSON Schema, typed model, or provider feature that constrains the response when available.
3. Validation: parse the result and check types, required fields, permitted values, and business rules.
4. Recovery: handle refusals, malformed output, missing information, timeouts, and schema changes without silently corrupting data.

Structured output and function calling are related but not identical. Structured output is generally aimed at returning data that matches a declared shape. Function calling, also called tool calling by some providers, is designed for selecting a tool and producing arguments for it. Both can be useful, but neither removes the need for validation. Treat model output as untrusted input, just as you would treat a request from an external client.

Checklist by scenario

Scenario 1: Extracting fields from text

For entity extraction, classification, or document parsing, start with the smallest useful schema. Define field names, types, required status, and what should happen when the source text does not contain an answer.

{
  "type": "object",
  "properties": {
    "customer_name": { "type": ["string", "null"] },
    "issue_type": {
      "type": "string",
      "enum": ["billing", "technical", "account", "other"]
    },
    "summary": { "type": "string" },
    "confidence": { "type": "number", "minimum": 0, "maximum": 1 }
  },
  "required": ["customer_name", "issue_type", "summary", "confidence"],
  "additionalProperties": false
}

• Use null when a field may be unknown; do not force the model to invent a value.
• Use enums for closed categories, but include an “other” option when the domain is open-ended.
• Keep confidence separate from truth. A model-generated confidence value is a signal for review, not proof.
• Specify whether dates, identifiers, and amounts must use a particular representation.

Scenario 2: Returning data for application code

When a backend consumes the response, validate it immediately at the API boundary. In Python, a typed validation library can make this explicit:

from pydantic import BaseModel, Field
from typing import Literal

class Ticket(BaseModel):
    customer_name: str | None
    issue_type: Literal["billing", "technical", "account", "other"]
    summary: str = Field(min_length=1)
    confidence: float = Field(ge=0, le=1)

parsed = Ticket.model_validate(model_json)

The exact library and syntax can differ between projects. The important pattern is to parse first, then pass the typed object to business logic. Do not let a raw model-generated dictionary flow directly into database writes, email delivery, or external tool execution.

Scenario 3: Asking the model to choose an action

Use function calling or tool calling when the model must select among explicit operations, such as lookup_order, create_ticket, or request_human_review. Keep tools narrow and define strict argument schemas. A tool call should describe an intended action; your application should still authorize it, validate its arguments, and decide whether confirmation is required.

A safe flow is: receive the proposed tool call, validate the arguments, check user permissions and current state, optionally request confirmation, execute the operation, and return a controlled result to the model. Never treat the presence of a valid function name as authorization to perform that function.

Scenario 4: Building a multi-step AI workflow

For workflows such as retrieval-augmented generation, split the process into contracts. One step can return retrieved passages, another can classify them, and a final step can produce the user-facing answer. Each boundary should have its own schema and tests. This makes failures easier to locate than one large prompt that asks the model to retrieve, reason, format, and act in a single response. For related workflow design, see this RAG guide for developers.

What to double-check

Provider and model capabilities

Provider support for JSON mode, schema-constrained output, function calling, streaming, refusals, and nested schema features can vary by API, model, SDK version, and request configuration. Before implementation, consult the current documentation for the exact model you plan to use. Confirm which schema keywords are supported, whether strict validation is available, how refusals are represented, and whether streaming changes the response-handling process.

A practical comparison should record the following rather than relying on a generic feature list:

Prompt and schema alignment

Your prompt should describe the task and the meaning of ambiguous fields, while the schema should enforce machine-checkable constraints. Avoid putting contradictory instructions in both places. For example, if the schema requires an enum, explain how to choose among those values rather than asking for a free-form category elsewhere.

Tell the model what to do when evidence is missing: return null, use a permitted fallback, or mark the item for review. Also state whether the response must contain only the structured object. This reduces, but does not eliminate, extra prose around the payload.

Validation and observability

Record enough information to diagnose failures without storing sensitive content unnecessarily. Useful fields include schema version, model identifier, request correlation ID, validation result, retry count, latency, token usage when available, and a redacted error category. Keep separate metrics for transport errors, provider errors, refusals, parse failures, schema failures, and business-rule failures. These indicate different fixes.

Common mistakes

• Using prompt instructions as validation: “Return valid JSON” does not replace a parser and schema validator.
• Retrying every failure identically: A timeout, refusal, invalid enum, and missing source evidence need different responses.
• Repairing JSON without boundaries: A repair pass may fix syntax but change meaning. Validate the repaired result and cap the number of attempts.
• Accepting unknown fields silently: Extra fields can indicate prompt drift, model changes, or an injection attempt. Decide whether to reject, strip, or quarantine them.
• Making schemas too broad: A response with many optional strings may be technically valid but operationally useless. Prefer precise types and explicit nullability.
• Making schemas too rigid: Requiring information that may not exist encourages fabricated values. Design an intentional unknown state.
• Mixing schema and business validation: JSON Schema can check shape, but it may not know whether an account exists or whether a requested action is allowed. Apply domain rules after parsing.
• Ignoring partial responses: Network interruptions and streamed output can leave incomplete JSON. Treat incomplete data as incomplete, not as a successful empty result.

A bounded recovery pattern is usually safer than an open-ended loop: attempt the original request, parse and validate, retry once with a concise error summary if the failure is recoverable, then either run a carefully limited repair step or route the item to a fallback path. Preserve the original response for debugging, but do not use it as trusted application data.

When to revisit

Revisit this checklist before a planning cycle, after changing an LLM provider or model, when upgrading an SDK, or whenever you add or remove schema fields. Also review it when a workflow begins handling new languages, longer documents, sensitive data, tool execution, or higher volumes. These changes can expose assumptions that were invisible in a small prototype.

Keep a small regression set containing normal cases, missing information, ambiguous wording, boundary values, malformed source text, refusal-prone requests, and adversarial instructions embedded in input. Run it whenever the prompt, schema, model, parser, or retry policy changes. Compare not only whether JSON parses, but whether the values remain useful and safe for the next step.

1. Version the schema and store the version with each processed result.
2. Write contract tests for required fields, enums, null handling, and unknown properties.
3. Test refusal, timeout, partial-response, and validation-failure paths explicitly.
4. Review retry limits, logging, privacy controls, and tool authorization.
5. Check the provider’s current documentation before relying on a newly advertised capability.
6. Deploy changes gradually and monitor validation failures separately from ordinary application errors.

For broader model selection and production-readiness criteria, pair this checklist with our guide to evaluating an LLM API. The durable principle is simple: use the model for interpretation, use schemas and validators for contracts, and use application code for authorization and final decisions.