"Return your answer as JSON" followed by JSON.parse() is one of the most common — and most fragile — patterns in LLM apps. The model adds a preamble, wraps it in a code fence, trails a comment, and your parser throws in production. Structured outputs fix this by constraining the model to a schema instead of hoping it behaves.
Why prose-parsing fails
When you ask for JSON in plain text, you're relying on the model to:
- emit only JSON (no "Here's your result:"),
- close every bracket and quote,
- use the exact field names you expect,
- never hallucinate an extra field.
At low volume it mostly works. At production volume, "mostly" means a steady trickle of 500s and silent data corruption. Regex cleanup is a losing battle.
The fix: constrain to a schema
Define the shape you want and have the framework enforce it — either via a JSON schema passed to the model or a typed mapping that validates the response. You stop parsing prose and start receiving objects.
// Spring AI — map the reply straight onto a record, no manual parsing
record Ticket(String summary, String severity, boolean escalate) {}
Ticket t = chatClient.prompt()
.user("Triage this support message: " + message)
.call()
.entity(Ticket.class);entity() instructs the model to produce output matching the record and deserializes it for you. The same idea applies via explicit JSON schema if you need finer control (enums, required fields, nested objects).
Make it robust
1. Be explicit about the contract. Describe each field and its allowed values. severity: one of "low" | "medium" | "high" beats a free-form string.
2. Use enums, not free text, wherever the value is from a fixed set — it removes a whole class of drift.
3. Validate after deserializing. Schema conformance ≠ semantic correctness. Check business rules (e.g. an escalated ticket must have a reason) in code.
4. Keep temperature low for extraction/classification — you want determinism, not creativity.
5. Plan for refusal. The model may legitimately have no answer. Model that explicitly ("found": false) rather than forcing a fabricated object.
Common pitfalls
- Over-nesting. Deeply nested schemas are harder for the model and for you. Flatten where you can.
- Optional everything. If every field is optional, you'll get sparse, inconsistent objects. Mark what's required.
- Mixing prose and data. If you need both an explanation and structured data, ask for the explanation as a field, not alongside the JSON.
Wrap-up
Structured output turns the model from a text generator into a reliable component you can wire into typed code. Define the schema, keep temperature low, validate the semantics, and delete your regex cleanup — it's the difference between a demo and a dependable feature.
Related reading
- Tool Calling with Spring AI — structured inputs to your services, the mirror image of this.
- Testing AI Applications — how to catch output drift before it ships.
- Context Engineering: The Real Skill Behind Reliable LLM Apps — what the model sees still decides quality.