Chapter 8: Batch and asynchronous inference

Overview#

A feedback-analysis pipeline classifies thousands of comments every night. Nobody reads the report until morning, yet the application may use the same real-time API mode as a customer-facing assistant.

Some providers discount deferred inference, but keep three things apart: asynchronous execution, provider Batch APIs and GPU-level inference batching.

A background worker calling the standard API is asynchronous to your user but may still pay standard token prices. A provider Batch API can offer a lower pricing tier for nonurgent requests. Continuous batching inside a serving engine improves throughput and has nothing to do with API discounts.

Practical#

Start with the product’s deadline. Interactive support often needs seconds; historical document processing may tolerate hours. Dependent agent steps limit what you can defer, because a downstream action can’t be submitted until the previous step produces its inputs.

Google, OpenAI and Anthropic each document a discounted batch mode for requests that can wait, with results returned within a turnaround window. As of this edition, Google lists a 50% price reduction for eligible Gemini Batch API use and a 24-hour turnaround target. Discounts, windows and eligible models differ by provider and change over time, so check the current terms for each model and endpoint before relying on them.

At a hypothetical standard price of $0.012 per task, 500,000 tasks cost $6,000. An eligible batch mode with equivalent output at half price costs $3,000 in inference charges. Net savings can fall short of $3,000 once you count job management, failed or expired requests, resubmissions and any work that still needs immediate execution.

Track every batch item on its own, so a partly failed job can be repaired without resubmitting the whole batch. The Deep Dive below covers the details.

Ledgerline. A quarter of Ledgerline’s invoices arrive in bulk monthly imports that nobody reads until the next morning. Those now go through the provider’s batch mode at half price, with each invoice tracked by its own ID. The saving is 25% of $648 at half price, about $81, and extraction settles at $567 a month.

Deep Dive#

For self-hosted inference, batching is a throughput technique: the serving engine interleaves requests of different lengths so the GPU stays busy. The gain depends on the workload, KV-cache pressure and latency budget, and GPU inference and deployment economics covers the mechanics.

Better throughput lowers the cost allocated to each task, but the capacity bill stays the same unless the change lets you reduce or avoid capacity.

Batching also combines with caching and deduplication. If only 5% of yesterday’s documents changed, skipping the unchanged ones may save more than the batch pricing discount.

Running provider batch jobs#

Before submitting, write each item to your own store with its application ID, a hash of the input, the model and prompt version, the provider’s job ID and a status such as pending, submitted, succeeded, failed or expired. Put the application ID in the request’s custom identifier field when the API offers one, and use it to join results back to your records. A result file may come back in a different order from the input, and some items may be missing from it.

A finished job can contain successful items, items that returned an error and items that expired before the provider processed them. Handle each one on its own. A malformed request or an input over the context limit will fail again unchanged, so fix it or drop it. Expired items, rate-limit errors and transient server errors are worth resubmitting.

Put the retryable items in a new job that carries the original application IDs, and leave the successful ones alone. Cap the attempts per item, for example two resubmissions, then move anything still failing to a review queue. Without a cap, a systematic fault such as a prompt template that always exceeds the context limit keeps generating new jobs.

Make downstream processing idempotent. A worker can crash after reading a result file and read it again, and an item you resubmitted after a timeout can still produce a result from the first attempt. Write results with an upsert keyed on the application ID and prompt version, and have consumers skip IDs they have already handled. A duplicate result then costs tokens but doesn’t create a second ticket, email or database row.

Count everything the job cost. Include charges for items whose output you discarded, charges for every resubmission and any items you moved to the real-time API at standard prices. Divide that total by the number of items that produced a usable result. That figure is the one to compare with real-time processing, and with a high failure rate it can sit well above the headline batch price.

Decide what happens to items that miss the deadline before the first job runs. If the morning report is due at 08:00 and the provider’s window allows up to a day, a job submitted at midnight may not finish in time. The options are to send late items to the real-time API, publish the report with those items marked as pending, or skip them when the data will be stale anyway. Each option has a different cost, so the product owner should pick one and the pipeline should implement it explicitly.

References: Gemini Batch API, Gemini optimization modes, OpenAI Batch API, Anthropic batch processing.