Cost and latency tradeoffs in LLM system design
Quality, cost, and latency behave like the classic project-management triangle: pick two, and the third moves. A design that never names its cost and latency budget hasn't avoided the tradeoff. It has let whichever model and call pattern the code defaulted to make the decision.
1. Every request has an implicit budget
A feature that calls a model on every request makes a cost and latency decision whether or not anyone writes a number down. The model chosen, the amount of context sent, and whether the response streams or waits for completion add up to a per-request cost and a per-request latency. With no explicit budget, the system inherits whatever the defaults produced. Name both numbers up front for each feature: how much latency this interaction tolerates before it feels broken, and what per-request cost is acceptable at the expected volume. Those two answers decide which levers you can use.
2. Model tiering: routing by task difficulty
Sending every request to the largest available model is the common first-pass design and rarely the right steady state. Extended thinking: architecting for Claude's reasoning mode makes the same argument against defaulting deep reasoning onto every request. A simple classification, a short rewrite, or a lookup with a clear answer gets nothing from the largest model that a smaller, faster, cheaper one wouldn't deliver as reliably. Paying the larger model's cost and latency on each of those requests compounds across volume and buys no quality.
Model tiering routes each request to the model that matches its difficulty: a lightweight model for routine, well-defined tasks, escalating to a larger model only for requests that benefit from it. The routing logic is the same kind section 3 of the extended-thinking guide lays out: a cheap upfront classifier, a fixed rule by request type, or escalate-on-failure. Model tier and extended reasoning are separate levers that follow one principle: match the resources spent to the difficulty of the request.
3. Caching as a cost and latency lever together
Prompt caching architecture is usually framed as a cost optimization. It is a latency lever too: a cached portion of a prompt isn't reprocessed, which cuts time to first token as well as the cost of the call. A system that sends a large static system prompt or a repeated set of few-shot examples on every call pays a cost tax and a latency tax on that content every request. Structure the prompt so the static portion is cache-eligible (see that guide's section on what counts as "static enough") and one design change captures both savings.
4. Streaming for perceived latency, not actual latency
Streaming architecture for Claude applications reduces perceived latency (how long a user feels they're waiting) without necessarily reducing total generation time. A streamed response that takes as long as a non-streamed one feels faster because content appears immediately. That is valuable for interactive, user-facing features. It does nothing for cost or total processing time, which matters when picking the lever. A background job with nobody watching gets nothing from streaming.
5. Batching for throughput, not latency-sensitive work
Batch processing architecture: the Message Batches API is the lever at the other end from streaming: work with no per-item latency requirement, where the goal is processing a large volume efficiently. Batching a latency-sensitive interactive feature gives up the one thing that use case needs. Running a large offline job as individual synchronous calls is the opposite mistake: you pay interactive pricing and rate-limit structure for work that never needed fast turnaround. Most of this decision is correctly placing the workload on one side of that line before reaching for either lever.
6. The tradeoff triangle, stated plainly
None of the four levers escapes the tradeoff between quality, cost, and latency. Each one moves where part of it gets paid, or removes a cost that wasn't buying any quality. Routing an easy request to a smaller model costs no quality if the task didn't need the larger model. Caching removes a repeated cost for static content that gained nothing from being reprocessed. Both stop you overpaying for quality the system wasn't using. A harder task, where the larger model's extra capability is doing the work, still costs more and takes longer, and no caching or routing changes that.
7. Worked example: redesigning a document-summarization feature
A document-summarization feature calls the largest available model synchronously on every request. The full document goes into every call, and the user sees nothing until the complete summary is ready. In practice most documents are short and easy to summarize. A smaller fraction are long, complex, or technical enough that the larger model's extra capability matters. The system pays the larger model's full cost and latency on every request, easy and hard alike.
The redesign applies each lever where it fits:
- A cheap upfront length-and-complexity check routes short, straightforward documents to a smaller, faster model and escalates only the documents that need the larger one.
- The response streams whichever model handled it, so the user sees the summary building immediately.
- A separate internal use case, a first-pass summary for every document in a large historical archive with nobody waiting on any one result, moves off the synchronous per-document pattern onto a batch job. That cuts its cost and stops it competing for the interactive-latency infrastructure the user-facing feature needs.
None of these changes lowers the quality of a summary the larger model was earning its cost on. They stop paying that cost on requests and workloads that weren't benefiting from it.
For the mechanics of each lever, see prompt caching architecture, streaming architecture for Claude applications, and batch processing architecture. For the related routing decision of when deeper reasoning, as opposed to a larger model, is worth its cost, see extended thinking: architecting for Claude's reasoning mode.