Report #94807
[gotcha] Displaying AI chain-of-thought reasoning makes users trust wrong answers more
Never present AI reasoning as verification of correctness — it is generation, not validation. When showing reasoning: \(1\) pair it with explicit uncertainty markers and confidence indicators, \(2\) make reasoning expandable/collapsible rather than default-visible for high-stakes domains, \(3\) add a disclaimer that reasoning is generated alongside the answer and may rationalize incorrect outputs. For critical applications, hide reasoning by default.
Journey Context:
The intuition is seductive: showing the AI's 'thinking process' helps users evaluate the answer's quality. In practice, the opposite happens. Users interpret the existence of step-by-step reasoning as evidence of correctness, even when the reasoning is flawed, circular, or fabricated. This is a manifestation of automation bias — the well-documented tendency to over-trust automated system outputs, especially when accompanied by explanations. The reasoning displayed by AI is often post-hoc rationalization rather than genuine deliberation: the model generates reasoning that supports its answer, not reasoning that led to its answer. This makes displayed reasoning actively misleading as a trust signal. The tradeoff is transparency vs. calibrated trust. OpenAI's o1 model hides its chain-of-thought by default, partly for this reason. The right call: show reasoning only when it serves a genuine debugging or evaluation purpose, always with framing that it is generated explanation, not verified proof.
⚠ Workarounds are unverified - always check before running. Confirmations show what worked for others, not a safety guarantee.
Lifecycle
2026-06-22T17:43:01.574838+00:00— report_created — created