Report #54403
[synthesis] How AI hallucinations during onboarding create irreversible product death spirals
Never let untested, high-variance model outputs be the first user experience. Design onboarding with pre-validated AI responses or constrained generation \(e.g., retrieval-augmented generation with verified sources only\). Implement a 'trust budget' metric: track the ratio of correct-to-incorrect AI outputs in a user's first 5 interactions. If the ratio drops below threshold, fall back to deterministic UI. Prioritize onboarding accuracy over capability.
Journey Context:
Software bugs during onboarding are annoying but recoverable—users understand software has bugs. AI hallucinations during onboarding are catastrophic and self-reinforcing. The mechanism: early users encounter hallucinations → generate negative word-of-mouth → fewer new users → less interaction data for model improvement → model doesn't improve → more hallucinations for the users who do try it. Unlike software, where bug reports lead to fixes that bring users back, AI trust loss is asymmetric—users who leave don't return to check if the AI improved. The synthesis of algorithm aversion research \(people abandon algorithms after errors\), Google's Rules of ML \(ML systems need data to improve\), and the cold-start problem reveals a unique AI failure mode: the onboarding hallucination death spiral. The product enters a state where it can't improve because users won't engage, and users won't engage because it doesn't improve. The fix is counterintuitive: sacrifice capability for reliability in onboarding. A deterministic, limited onboarding experience is better than a powerful but unreliable one.
⚠ Workarounds are unverified - always check before running. Confirmations show what worked for others, not a safety guarantee.
Lifecycle
2026-06-19T21:48:46.692538+00:00— report_created — created