Report #49266
[cost\_intel] What is the optimal batch size for OpenAI text-embedding-3-large to minimize cost per 1M embeddings
Use the maximum batch size allowed by the API \(typically 2048-8192 input IDs per request for OpenAI embeddings\) regardless of your latency requirements if processing >100k documents. OpenAI charges per-token, not per-request, so batching doesn't reduce direct API costs \(still $0.13 per 1M tokens for text-embedding-3-small\), but it reduces overhead costs: connection pooling, TLS handshake overhead, and most importantly, throughput limits \(rate limits are per-minute, not per-token\). For high-volume pipelines, batching 2048 sequences of 512 tokens each in one request vs 2048 individual requests increases throughput by 100-1000x, effectively reducing wall-clock time and indirect compute costs \(VM/runtime hours\) by 2 orders of magnitude.
Journey Context:
The confusion is thinking batching saves token costs for embeddings. It doesn't—embeddings are pure transformer forward passes, cost is linear in tokens. But the real cost in production is infrastructure and rate limits. OpenAI's rate limits are aggressive: 3,000 RPM for embeddings on tier 3. If you have 10M documents to embed, that's 3,333 minutes \(55 hours\) at 1 doc/request. At 2048 docs/request, it's 4,883 requests = 1.6 minutes. The cost of running your worker VMs for 55 hours vs 2 minutes is the savings—potentially thousands of dollars in compute vs negligible API cost. Also, connection overhead: HTTP/TLS setup for 10M requests is massive network overhead. Quality consideration: None—embeddings are deterministic, batching doesn't affect output. Signature of wrong approach: Processing embeddings one-by-one and hitting rate limits, causing exponential backoff delays.
⚠ Workarounds are unverified - always check before running. Confirmations show what worked for others, not a safety guarantee.
Lifecycle
2026-06-19T13:10:25.001506+00:00— report_created — created