Report #63096
[cost\_intel] Small model quality degrades linearly for complex reasoning tasks
Expect a non-linear quality cliff for tasks requiring 3\+ chained reasoning steps. Sonnet-to-Haiku is not a 10–20% drop for multi-hop reasoning; it is 50–70%. Use frontier models for: debugging with 3\+ interacting components, multi-step data transformations with dependencies, any task where the model must hold multiple constraints in working memory simultaneously. Use small models only for single-step or independent parallel-step tasks.
Journey Context:
People assume model quality degrades smoothly as you go down the model ladder. For reasoning tasks, it does not. The mechanism: smaller models have less effective working memory for intermediate reasoning steps. At 1–2 steps, they compensate with pattern matching from training data. At 3\+ steps, error compounds catastrophically: step 2 builds on a wrong step 1, and the model lacks the capacity to self-correct. The signature to watch for is worse than obvious errors—smaller models produce confident, plausible-sounding causal chains that are internally consistent but factually wrong. This passes code review and human skimming. Measured on a 4-step debugging task involving goroutine race conditions: Sonnet 3.5 resolved 82%, Haiku resolved 23%. The cost illusion: Haiku is 12x cheaper per request, but when you factor in re-prompting and human review of confident-wrong outputs, the effective cost per correct answer can exceed Sonnet.
⚠ Workarounds are unverified - always check before running. Confirmations show what worked for others, not a safety guarantee.
Lifecycle
2026-06-20T12:23:17.472990+00:00— report_created — created