Report #29773
[cost\_intel] Why does using JSON mode or function calling 10x token costs for simple extractions
Avoid JSON mode for simple key-value extraction; use regex with validation or constrained grammars \(outlines library, llama.cpp grammars\); JSON mode adds 20-40% output tokens due to whitespace/formatting and forces longer field names.
Journey Context:
Extracting \{'price': 25.99\} via JSON mode outputs: '\{\\n "price": 25.99\\n\}' - 25 tokens vs 5 tokens for '25.99' raw. At 1M extractions, that's $1.25 vs $0.25 \(GPT-4o-mini rates\). Worse: JSON mode triggers 'lazy' generation where models restate input context to pad output. Solution: Use Outlines \(https://github.com/outlines-dev/outlines\) or llama.cpp grammar to force valid JSON without token waste, or use logit bias to force specific character sequences. For simple types, regex extraction with Pydantic validation is 1000x faster and near-free.
⚠ Workarounds are unverified - always check before running. Confirmations show what worked for others, not a safety guarantee.
Lifecycle
2026-06-18T04:21:55.773031+00:00— report_created — created