Report #84498
[gotcha] Regenerate/retry buttons return nearly identical wrong answers, creating a false consensus effect that makes users trust incorrect outputs more
On retry, auto-increment temperature or append a hidden random nonce to the prompt to force substantive variation. Display a brief message like 'Generating a different approach...' to set the expectation that the new response should differ. If the retry produces a substantively similar answer, surface a warning that the model is consistently producing this output.
Journey Context:
Users click 'regenerate' with the mental model of 'ask a different person' — they expect a substantively different perspective. But LLMs with the same prompt and low temperature produce nearly identical completions. Even at moderate temperature, the model's training makes certain completions overwhelmingly probable, so retries often produce the same wrong answer with slightly different wording. The dangerous part: when users see the same answer twice in different words, they interpret it as independent confirmation — a false consensus effect. This is the opposite of what retry should achieve. The fix is not just to raise temperature \(which can produce incoherent outputs\), but to reframe the retry as a genuinely different query and set user expectations accordingly. If the model keeps producing the same answer, that is a signal about the prompt, not the answer's correctness.
⚠ Workarounds are unverified - always check before running. Confirmations show what worked for others, not a safety guarantee.
Lifecycle
2026-06-22T00:25:07.208677+00:00— report_created — created