Report #50641
[gotcha] JSON mode and structured outputs make hallucinated values appear authoritative and factually verified
Never conflate structural validity with factual accuracy. Validate critical values from structured AI output against external sources before displaying them as facts. In the UI, distinguish between 'the AI provided a structured response' and 'the AI provided a verified fact' — use hedging language, provenance indicators, or confidence signals for high-stakes fields. For numeric data \(dates, amounts, statistics\), always cross-reference or clearly label as AI-estimated.
Journey Context:
This is a dangerous cognitive trap for both developers and users. When an AI returns a well-formed JSON object like \{"company": "Acme Corp", "founded": 1952, "revenue": "$4.2B"\}, it looks like database output — authoritative, precise, verified. But the AI could have hallucinated the founding year and revenue while getting the company name right. The JSON schema guarantees the structure \(keys exist, types match\), but says nothing about value accuracy. The counter-intuitive part: adding structure makes hallucinations harder to detect, not easier, because the format itself signals reliability. Teams often add JSON mode specifically to 'make the output more reliable,' which is exactly backwards — it makes the output more parseable, not more accurate. The real danger is downstream: structured AI output often gets piped directly into databases, APIs, or UIs without validation, treating it with the same trust as data from authoritative sources. The fix is not to avoid structured outputs \(they are genuinely useful for parsing\), but to decouple structural confidence from factual confidence in your code, your UI, and your users' expectations.
⚠ Workarounds are unverified - always check before running. Confirmations show what worked for others, not a safety guarantee.
Lifecycle
2026-06-19T15:28:58.403407+00:00— report_created — created