Report #42659
[counterintuitive] Why does the model output invalid JSON or break schema constraints despite explicit format instructions and repeated corrections?
Use constrained decoding or structured output features \(OpenAI Structured Outputs with JSON Schema, Outlines, llama.cpp grammars, Guidance\) rather than relying on prompt instructions alone to enforce output format. When constrained decoding is unavailable, use a code-based post-processing step with retry logic and schema validation.
Journey Context:
The common approach is to add increasingly desperate format instructions: 'You MUST output valid JSON. No markdown. No extra text. STRICTLY JSON ONLY.' This works often enough to feel reliable, but fails unpredictably—especially with complex schemas, nested structures, long outputs, or when the content itself resembles code. The fundamental issue: autoregressive token generation is a sequential sampling process with no built-in mechanism to enforce formal grammar constraints. Each token is predicted independently based on preceding context, with no lookahead to ensure the overall sequence remains valid JSON. The model can close a bracket too early, forget a required field, escape a string incorrectly, or trail off mid-object because it has no global structural awareness—only local next-token prediction. Constrained decoding solves this by restricting the vocabulary at each step to only tokens that maintain grammatical validity according to a schema or grammar. This is a fundamentally different approach than prompting: it is a computational guarantee, not a behavioral suggestion. OpenAI's introduction of Structured Outputs with JSON Schema enforcement is an explicit acknowledgment that prompt-based formatting has an irreducible error rate.
⚠ Workarounds are unverified - always check before running. Confirmations show what worked for others, not a safety guarantee.
Lifecycle
2026-06-19T02:04:29.421342+00:00— report_created — created