Streaming architecture for Claude applications

1. Why streaming is an architecture decision

Streaming cuts perceived latency sharply. A user watching the response appear token by token, instead of a blank state until it's all ready, gets a better experience on long responses even though total generation time is the same. The cost is architectural: every piece of code downstream of the model call that assumed a complete, parseable response must be written or rewritten to handle partial fragments arriving over time.

2. What comes across the wire

A streamed response arrives as a sequence of events, typically server-sent events, each carrying a delta: the new piece of the response, not the response so far. Text streams as a sequence of text deltas. To display the full text so far, the client concatenates deltas itself; no event carries the accumulated text.

Anything about the response as a whole (its total length, whether it contains some content, why generation stopped) isn't fully known until the stream completes. Code that needs one of those answers must wait for the end of the stream or track the relevant state incrementally. The first event doesn't carry it.

3. Partial JSON: the specific hazard of streaming structured output

When a call streams structured output, most concretely a tool call's arguments, the JSON arrives in fragments like text does. For almost the whole stream, the accumulated string is a truncated fragment and not valid JSON until the closing braces and brackets arrive. Code that calls JSON.parse on the accumulated string at every delta, hoping to show a live, partly filled view, fails on nearly every intermediate attempt and succeeds only at the end, which defeats incremental rendering.

Use a partial-JSON parser, built to tolerate an incomplete document and return whatever structure it can extract with confidence from a truncated fragment (a field that closed cleanly even though the object hasn't). That's what lets a form's fields fill in one at a time as an extraction streams, or a proposed action's arguments appear progressively, instead of showing nothing until the end or crashing on invalid intermediate JSON.

4. Tool calls under streaming

A tool call can't execute until its arguments are complete. Calling a function with a half-formed argument object is wrong. Execute nothing against a tool call's arguments until the stream signals that call is finished. Track, per tool call, whether it's still accumulating or complete, and gate execution strictly on completion. The UI can still show the same call's arguments filling in, using the Section 3 partial-JSON handling for display only.

Keep "safe to show progressively" and "safe to execute" separate in the code. A bug that ties tool execution to the partial state driving a progressive UI update will eventually execute a tool call with incomplete, wrong arguments.

5. Cancellation and backpressure

Users navigate away, close tabs, and cancel long streamed responses mid-generation. This is normal. Without an explicit cancellation path, the model call runs to completion and you're billed for output nothing will consume. Propagate cancellation from the client through your backend to the model call, so a closed connection stops generation instead of only stopping the client from listening. That plumbing is easy to skip in a first implementation and expensive to have skipped once traffic and the cost of cancelled generations grow.

Backpressure is the less obvious half. A slow consumer (a client on a poor connection, a backend process that can't keep up with the delta rate) needs the stream to buffer safely or push back toward the source. Otherwise deltas pile up in an unbounded buffer or get dropped, and either way the consumer sees something different from what was generated.

6. Streaming through your own backend versus proxying directly

Streaming straight from the model API to the browser is the simplest architecture and the hardest to extend. There's no server-side validation before the client sees content, no place to enforce a guardrail (see guardrails architecture) on output the client is already rendering, and no natural point to log or persist the complete response, because the backend was never in the data path.

Streaming through your own backend, which relays deltas to the client while accumulating them server-side, adds value and complexity. A guardrail can inspect content as it streams and, in principle, interrupt a stream producing something it shouldn't. The final response is captured for logging or storage without another round trip. Cancellation and backpressure are handled consistently in one place instead of every client reimplementing them against the model API. The costs are some added latency from the extra hop and more complex infrastructure to build and keep correct. That's worth it for a production system with guardrail and logging requirements and much less clearly worth it for a low-stakes prototype.

7. Worked example: rendering a streamed tool-call argument safely

An application lets Claude propose a calendar event by calling a create_event tool, and the product wants the event's fields (title, time, attendees) to fill in visibly as the model generates the call. The naive implementation runs JSON.parse on the accumulated argument string at every delta and catches the exceptions. It mostly works, but occasionally flashes stale or half-wrong data when a delta lands mid-value, after a field name closed and before its value finished. Parse-and-catch can't tell that state apart from other partial states.

The fix uses a dedicated partial-JSON parser that returns only the fields that have unambiguously closed. A field mid-value is absent from the partial object, so the UI shows title and time appearing cleanly as each completes, with no flicker. Separately, and regardless of what the UI shows, the create_event call is gated on the stream signaling that this tool call is complete, so no event is ever created from an argument object the UI would still show as filling in.

See claude tool use and function calling for the tool-call mechanics being streamed here, and structured data extraction for the validation a streamed structured output still needs once it's complete.