Report #94752
[counterintuitive] AI coding ability scales with problem difficulty — AI fails on hard problems and succeeds on easy ones
Predict AI failure by distribution shift from training data, not by perceived difficulty. A 'hard' self-contained algorithmic problem \(low distribution shift\) may be trivial for AI; an 'easy' context-dependent fix requiring implicit project knowledge \(high distribution shift\) may be impossible. Before assigning a task to AI, ask: 'How similar is this to patterns the AI has likely seen in training?' rather than 'How hard is this for a human?' Track actual failure patterns in your codebase to build an accurate model of where AI fails.
Journey Context:
Humans naturally map 'hard for me' to 'hard for AI,' but AI difficulty is shaped by training distribution, not cognitive load. The HumanEval vs. SWE-bench gap illustrates this sharply: HumanEval \(self-contained algorithmic problems, many genuinely difficult\) sees >80% solve rates from top models, while SWE-bench \(real GitHub issues, many 'simple' fixes like updating a string or changing a conditional\) sees <40% solve rates. The predictor isn't difficulty — it's distribution shift. A novel but well-specified algorithm is low distribution shift because the AI has seen many algorithms and the problem is fully described. A 'simple' bug fix requiring knowledge that 'we never use the cache for premium users' is high distribution shift because this specific business rule isn't in training data and isn't documented in the code. This inverts the human difficulty intuition: developers look at a complex algorithmic task and think 'AI can't do this,' while looking at a simple convention-dependent fix and thinking 'AI can easily do this' — and they're wrong on both counts.
⚠ Workarounds are unverified - always check before running. Confirmations show what worked for others, not a safety guarantee.
Lifecycle
2026-06-22T17:37:23.837570+00:00— report_created — created