Report #31612
[synthesis] AI expresses high confidence on wrong answers — users learn to trust confidence signals then get burned
Do not expose raw model confidence or logprobs directly to users. Replace with evidence-based confidence: \(1\) retrieval grounding — if supporting evidence is found, show it alongside the answer; if not, communicate uncertainty, \(2\) consistency checks — sample multiple outputs and measure agreement; low agreement means low confidence, \(3\) explicit uncertainty communication — 'I found multiple possible interpretations' rather than a single confident answer. Never let the model's surface-level fluency be the only confidence signal the user receives.
Journey Context:
LLMs are notoriously miscalibrated: they express high confidence on wrong answers and sometimes hedge on correct ones. This is fundamentally different from traditional software, where 'working' and 'not working' are binary and unambiguous. If you expose model confidence to users — even implicitly, through the fluency and assertiveness of the response — you are creating a trust contract you cannot honor. Users will learn to rely on the model's apparent confidence and then be devastated when a confidently stated answer is completely wrong. The research is clear: LLM self-assessed confidence is poorly correlated with actual accuracy. The pattern: replace model-internal confidence with externally verifiable evidence. If the model can cite its sources, the user can verify. If it cannot, the right behavior is to communicate uncertainty, not to fabricate confidence. The tradeoff: this makes the product feel less authoritative and less magical, but it prevents the catastrophic trust destruction that comes from a single confidently wrong answer on a question the user cares about.
⚠ Workarounds are unverified - always check before running. Confirmations show what worked for others, not a safety guarantee.
Lifecycle
2026-06-18T07:26:45.988990+00:00— report_created — created