Batch processing architecture: the Message Batches API
Use a synchronous call for anything a person is waiting on. Use a batch for a job that runs against thousands of records overnight, where it doesn't matter whether one result takes two seconds or twenty minutes. Switching to batch changes more about the pipeline than the API call.
1. The tradeoff batching exists for
A synchronous call optimizes for latency: the response comes back as soon as generation finishes, with pricing and rate limits built for interactive use. Batch processing optimizes for throughput and cost. You submit a set of requests together, they process asynchronously, possibly over a longer window than any single call would take, and in exchange you get better cost efficiency on the whole job and a much higher request ceiling than the same volume sent as individual calls.
The deciding question is whether any single result is time-sensitive, regardless of total volume. A thousand-record nightly job with nobody waiting on a specific record is the shape batching is built for. A thousand-record job where each result must return in a couple of seconds because a user is waiting is not a batching problem at any volume, because per-item latency is the one thing that use case can't give up.
2. What changes architecturally
The interaction model inverts. You submit a batch once, as a set of individually identified requests, then poll for the job's status and retrieve results when processing completes. The submission call does not return results. Calling code can't assume a result exists right after it triggers the work; it needs a job identifier, a completion signal (polling or webhook), and a separate retrieval step. That is a different shape of code from a synchronous call site with a longer timeout.
Items in a batch don't complete uniformly. The batch can report complete while individual items succeeded, failed, or were skipped for reasons specific to that item. Retrieval and downstream logic must treat a batch as a collection of items, each with its own outcome.
3. Good fits and bad fits
Good fits are large offline jobs: classifying or scoring a day's support tickets overnight, running structured extraction over a document backlog, generating a first-pass summary for every item in a large dataset. Nobody is waiting on any one item. Aggregate throughput and cost matter far more than per-item turnaround, which is the profile batching is priced and designed for.
Bad fits are anything interactive: a chat interface, an agent responding to a live user action, anything whose value depends on arriving during an ongoing interaction. Batching these adds latency and usually breaks the feature outright, because a chat or agent experience assumes the response returns on a timescale the interaction can absorb.
4. Result handling: partial completion and per-item retry
Check each item's status on retrieval. A completed batch does not mean every item succeeded. A failed item, whether from a transient issue or a bad input, must not drop silently out of the pipeline. Log it, then either retry it alone (in a new, smaller batch or as a synchronous call, depending on how urgent recovery is) or route it to wherever this pipeline sends failed items for review.
Retry only the failed subset. Resubmitting the whole batch because a fraction failed wastes the work that succeeded. Targeted re-runs depend on per-item tracking from submission through to result.
5. Designing the surrounding pipeline for resumability
The pipeline around a batch can fail too: the polling process crashes, or a deploy interrupts the job that retrieves and processes results. It must resume without losing completed work or processing items twice, so batch and item state belongs in durable storage, outside the memory of whichever process submitted the job. Make per-item processing idempotent: processing the same item's result twice must produce the same end state as processing it once. That property makes resume and retry safe without error-prone bookkeeping about what has already been applied.
This is the same idempotency discipline an event consumer needs for the at-least-once delivery that event-driven architecture describes, applied to batch results. Neither system can promise a consumer sees each unit of work exactly once, so build the pipeline to tolerate duplicates.
6. Combining batch with prompt caching
A batch that runs the same system prompt, extraction schema, or evaluation rubric across every item is exactly the shape prompt caching architecture is built for. The shared instructions and schema are strong caching candidates; only the per-item content varies. No special integration is needed. Apply caching's structuring rule (static, cache-eligible content first) to a workload that repeats the same static content across every request by construction.
7. Worked example: nightly ticket classification
A support platform wants every ticket closed during the day classified into a fixed taxonomy overnight for the next morning's reporting. No classification is needed in real time. Several thousand synchronous calls at end of day would work, but they cost more than necessary for a job with no latency requirement, and they're fragile: a transient failure partway through a long sequential loop means restarting from wherever the loop tracked progress, with nothing preventing reprocessing of items that already succeeded.
The batch redesign submits the day's tickets as one batch at close of business. The classification instructions and taxonomy, shared by every request, are structured for caching; each ticket's content is the per-item portion. A scheduled job polls for completion the next morning, retrieves results, and writes each classification idempotently, so a retrieval step that runs twice or restarts after a partial failure doesn't double-write. Tickets that failed are logged with their specific failure and retried individually. The nightly job is not marked failed because a small fraction of several thousand tickets failed on the first pass.
See prompt caching architecture for the caching design this repeated-instruction batch benefits from, and structured data extraction for the schema and validation layer a classification or extraction batch still needs after retrieval.