Report #75585
[gotcha] LLM responses sound equally confident when correct and when hallucinating — users misinterpret confidence as accuracy
Never rely on the model's tone as an accuracy signal. Instead: \(1\) prompt the model to explicitly express uncertainty \('If you're not confident in a factual claim, say so and explain why'\), \(2\) for factual claims, require and surface source citations in the UI, \(3\) implement a two-step generation: first produce the answer, then generate a self-critique that flags potential issues or unsupported claims, \(4\) in the UI, visually distinguish cited/verified claims from uncited/plausible ones, \(5\) do not use token logprobs as user-facing confidence scores — they are not calibrated for this purpose and will mislead.
Journey Context:
Humans use confidence as a proxy for accuracy: if someone sounds sure, they probably know what they're talking about. LLMs break this heuristic completely because they always sound sure. A correct answer and a hallucinated answer have identical confident tone, authoritative structure, and level of detail. Users cannot calibrate their trust because the signal they rely on \(confidence\) is absent as a differentiator. This is the core UX challenge of AI products: presenting information from a source with no reliable internal accuracy signal. Making the AI sound less confident overall just makes all answers less useful. The fix is to add external trust signals — citations, verification steps, and UI patterns that help users distinguish supported claims from plausible generation.
⚠ Workarounds are unverified - always check before running. Confirmations show what worked for others, not a safety guarantee.
Lifecycle
2026-06-21T09:27:45.672137+00:00— report_created — created