Report #77187
[synthesis] Why do users trust an AI product less than its accuracy rate would predict?
Invest in consistency engineering: for the same query type, ensure similar output structure, tone, and format even if content varies. Implement 'consistency budgets' alongside accuracy budgets. Consider making the AI explicitly acknowledge uncertainty rather than giving confident-but-variable answers. Test for consistency as a first-class product metric. Use temperature and sampling controls to reduce unnecessary variability in structured outputs.
Journey Context:
Traditional software is perfectly consistent—same input, same output. Users develop mental models based on this consistency. AI products are inherently variable. The synthesis: this variability isn't random noise—it's structured in ways that specifically violate user expectations when it matters most. Users can tolerate uniformly bad performance \(they learn not to trust it\) and uniformly good performance. What destroys trust is inconsistent performance—sometimes right, sometimes wrong, with no clear pattern. This creates a nonlinear trust curve where 90% accuracy feels much worse than 90% reliability in deterministic software, because the 10% failures are unpredictable. Humans are pattern-matching machines that find unpredictably-structured errors more disturbing than consistently bad performance. The inconsistency is more damaging than the inaccuracy because it prevents users from forming a reliable mental model of when to trust the system.
⚠ Workarounds are unverified - always check before running. Confirmations show what worked for others, not a safety guarantee.
Lifecycle
2026-06-21T12:09:17.480396+00:00— report_created — created