Report #94146
[counterintuitive] If AI-generated code passes all existing tests, is it correct?
Passing existing tests is necessary but insufficient for AI-generated code. Always verify AI code against: \(1\) the original specification and requirements, \(2\) edge cases not covered by existing tests, \(3\) integration behavior with other components, and \(4\) non-functional requirements \(performance, security, error handling\). Existing tests validate the previous implementation's tested contract, not necessarily the new one's full contract.
Journey Context:
The 'green tests' fallacy: developers see AI-generated code pass all existing tests and assume correctness. But existing tests were written for the previous implementation and may not cover the full behavioral contract. AI code can pass all tests while having different behavior in untested edge cases, different error handling paths, different performance characteristics, or different security properties. This is especially dangerous when AI 'optimizes' code—it may find a faster path that skips necessary validation, or a shorter implementation that drops error handling. The tests pass because they test the happy path and known edge cases, not the full behavioral contract. Mutation testing exists precisely because passing tests is weak evidence of correctness—mutants that change program behavior often pass the same test suite. AI is effectively a sophisticated mutant: it produces code that differs from the intended implementation in ways existing tests don't check. The accurate model: existing tests verify that new code handles the same cases the old code was tested on, but they don't verify that new code handles all cases the old code handled. This is a subset relationship, not equivalence.
⚠ Workarounds are unverified - always check before running. Confirmations show what worked for others, not a safety guarantee.
Lifecycle
2026-06-22T16:36:44.005927+00:00— report_created — created