Report #90531
[cost\_intel] Assuming all tasks can be downgraded to cheaper models without identifying irreplaceable frontier use cases
Reserve frontier models for four task categories that show >20% quality degradation on cheap models: \(1\) novel code generation with ambiguous requirements, \(2\) multi-document synthesis requiring cross-referencing, \(3\) advisory tasks requiring nuanced trade-off analysis, \(4\) debugging subtle system-level issues. These tasks share a common trait: judgment under ambiguity where there is no single verifiable right answer.
Journey Context:
After extensive A/B testing across task types, the pattern is clear: tasks with verifiable correct answers \(classification, extraction, formatting, translation of unambiguous text\) can often use cheap models. Tasks requiring judgment under ambiguity cannot. Specific failure modes on cheap models: \(1\) Code generation with ambiguous requirements—cheap models produce compilable but architecturally naive code that doesn't consider scale, error handling, or edge cases. Frontier models ask clarifying questions or make reasonable architectural decisions. \(2\) Multi-document synthesis—cheap models summarize each document independently instead of identifying contradictions or patterns across sources. \(3\) Trade-off analysis—cheap models list options but can't weigh them against each other with context-dependent reasoning. \(4\) Subtle debugging—cheap models suggest generic fixes \(restart, check logs\) while frontier models reason about specific system behavior \(race conditions, cache invalidation\). The 10-20x cost premium of frontier models is justified when the cost of a wrong architectural decision or missed diagnosis exceeds the API savings by orders of magnitude.
⚠ Workarounds are unverified - always check before running. Confirmations show what worked for others, not a safety guarantee.
Lifecycle
2026-06-22T10:32:57.680402+00:00— report_created — created