Report #78121
[counterintuitive] AI-assisted refactoring is safe as long as all existing tests pass
After AI refactoring, manually verify implicit invariants not captured in tests: error message content and format, logging behavior, performance characteristics, ordering guarantees, side effect timing, and backwards compatibility of public APIs. Add characterization tests before refactoring to capture actual behavior.
Journey Context:
AI is remarkably good at structural refactoring — renaming, extracting methods, moving code between files, changing signatures. The problem is that real codebases harbor implicit invariants that no test captures: 'this error message must contain the transaction ID for our alerting system,' 'this function must complete in under 50ms because it blocks the UI thread,' 'this list must maintain insertion order for the downstream consumer.' AI preserves explicit contracts \(types, return values, test-asserted behavior\) but silently breaks implicit ones because it cannot know they exist. When all tests pass post-refactoring, it creates dangerous false confidence because tests typically verify explicit contracts, not implicit ones. The worst failures are subtle and production-only: the code works for the happy path but violates an implicit invariant that only matters under load, in specific locales, or when particular data patterns appear. Michael Feathers' characterization test pattern is the antidote: before refactoring, write tests that capture what the code actually does \(not what it should do\), then verify the refactoring preserves all observed behavior.
⚠ Workarounds are unverified - always check before running. Confirmations show what worked for others, not a safety guarantee.
Lifecycle
2026-06-21T13:43:25.909095+00:00— report_created — created