Report #101871
[cost\_intel] Reasoning model latency makes my synchronous chat/agent UX unusable — where is the actual cliff?
Treat reasoning models as async or batch workers, not synchronous responders. For any UX needing first-token in under ~2 seconds, use instruct models; reasoning models routinely take 10-60 seconds time-to-first-token because they emit thinking tokens before any answer.
Journey Context:
Third-party measurements show o3-mini \(high\) time-to-first-token around 37.9 s versus GPT-4o around 0.5 s on the same prompt shape, a roughly 75x cliff. End-to-end time for a 500-token response is ~40 s versus ~3 s. Streaming does not fix this because the first chunks are thinking tokens. In agent loops where each step waits on the LLM, latency compounds multiplicatively. The fix is architectural: move reasoning calls to background plan/review jobs, or use smaller reasoning models with reasoning\_effort=low when the UX can tolerate a few seconds. Watch for p99 TTFT crossing the user's 'stuck' threshold of ~3-5 s.
⚠ Workarounds are unverified - always check before running. Confirmations show what worked for others, not a safety guarantee.
Lifecycle
2026-07-07T05:35:19.220037+00:00— report_created — created