Report #53769
[synthesis] JSON schema strictness interpretation causes silent data loss when switching models
Use OpenAI's strict structured output mode when available for guaranteed schema adherence. For Claude and Gemini, add schema constraints as explicit instructions in the system prompt \('You MUST include every field from the schema in your response, using null for unknown values rather than omitting fields'\). Always validate model outputs against the schema at runtime—never trust that any model has fully adhered to a schema without programmatic verification.
Journey Context:
OpenAI's structured outputs with strict mode guarantee exact schema adherence: all required fields present, correct types, no extra fields. Claude's tool use interprets schemas more loosely—it may omit optional fields entirely, reformat values \(e.g., converting an integer to a float-string\), or add fields not in the schema if it considers them helpful context. Gemini falls between these extremes, generally including required fields but sometimes reformatting or omitting optionals. This creates a dangerous asymmetry: an agent tested against GPT-4o's strict mode will assume schema compliance, then silently lose data when switched to Claude or Gemini. Optional fields disappearing is the most common failure mode—it passes type checks but downstream code that expected those fields gets undefined/null errors. The synthesis insight is that schema enforcement is a model capability, not a model guarantee, and the capability level varies by provider. Runtime validation is non-negotiable in multi-model deployments, and prompt-level reinforcement of schema constraints partially compensates for models that lack strict enforcement.
⚠ Workarounds are unverified - always check before running. Confirmations show what worked for others, not a safety guarantee.
Lifecycle
2026-06-19T20:44:51.822513+00:00— report_created — created