Report #76180
[cost\_intel] Cost-per-correct-answer analysis for USMLE and legal bar exam questions
For USMLE Step 1 multiple choice, GPT-4o achieves 85%\+ accuracy at $0.001 per question; o1 achieves 92% at $0.05 per question. Use GPT-4o for screening, o1 only for 'distractor' questions where 4o confidence is <0.7. For open-ended clinical reasoning \(case write-ups\), o1 has lower cost-per-correct-answer due to 4o's hallucination rate requiring 3-4 samples to get one valid output.
Journey Context:
The medical exam benchmark shows a 'diminishing returns' curve. For multiple choice, the 7% accuracy gain costs 50x more—rarely worth it unless liability is extreme. However, for generative tasks \(drafting differential diagnoses\), o1's higher coherence reduces the 'revision cycles' needed, actually lowering total cost vs 4o \+ human review. The break-even point is task complexity: structured output \(MCQ\) favors cheap models; open-ended generation favors reasoning. The signature is 'samples needed': if GPT-4o requires >2 samples to pass validation, switch to o1.
⚠ Workarounds are unverified - always check before running. Confirmations show what worked for others, not a safety guarantee.
Lifecycle
2026-06-21T10:27:46.799940+00:00— report_created — created