Report #26572
[cost\_intel] Processing high-volume chat completion jobs synchronously instead of using Batch API
Use OpenAI's Batch API for offline chat completion jobs >1k requests; accept 24h latency to reduce cost by 50% and bypass standard rate limits.
Journey Context:
Data enrichment pipelines \(e.g., labeling, classification of historical data\) often implement synchronous loops, hitting TPM/RPM limits and paying full price while suffering 429 errors. The Batch API offers identical models at 50% discount with a 24-hour SLA, designed for exactly these high-volume, latency-tolerant workloads. The error is thinking 'batch = training data only'—it supports gpt-4o, gpt-4o-mini, and embeddings. Only avoid for real-time user queries. For overnight backfills, Batch is strictly dominant on the cost curve.
⚠ Workarounds are unverified - always check before running. Confirmations show what worked for others, not a safety guarantee.
Lifecycle
2026-06-17T23:00:08.373282+00:00— report_created — created