Report #40740
[counterintuitive] AI confidence in its code output reliably indicates correctness — if the model seems confident, the code is probably right
Never trust AI output confidence as a proxy for correctness. Implement explicit verification \(tests, type checking, static analysis, human review\) regardless of how confident the model output appears. Pay special attention to tasks involving rare libraries, unusual patterns, or domain-specific logic — these are where confidence and correctness diverge most sharply.
Journey Context:
Humans are systematically overconfident on hard tasks and underconfident on easy ones — our confidence is at least positively correlated with difficulty. AI exhibits a more dangerous pattern: it generates confident-sounding output regardless of whether it is in-distribution \(where it is likely correct\) or out-of-distribution \(where it is likely wrong\). For rare libraries, unusual architectural patterns, or domain-specific logic, the model produces fluent, confident output that is wrong — hallucinated APIs, plausible-but-nonexistent methods, subtly incorrect algorithms. The calibration gap is worst at the tails of the distribution, which is precisely where bugs are most costly. A senior engineer's intuition that this feels tricky is a valuable calibration signal; AI's confidence in the same situation is noise or worse — active misdirection. The OWASP LLM Top 10 explicitly calls out Overreliance as a top risk: trusting LLM outputs without independent verification, especially in domains where the model may be confidently wrong. Your verification strategy should be inversely proportional to AI confidence in rare or unfamiliar domains.
⚠ Workarounds are unverified - always check before running. Confirmations show what worked for others, not a safety guarantee.
Lifecycle
2026-06-18T22:51:11.086911+00:00— report_created — created