Claude and retrieval: long context vs. RAG, and citations
Every application that grounds Claude in an organization's data has to decide how much of the corpus the model sees on each call. A large context window puts a bounded corpus directly in front of the model. Retrieval fetches only the relevant slice at query time. Neither is the better default; the shape of the corpus almost always decides.
1. Two different answers to "how does the model see my data"
Long context means the whole relevant document set, or as much as fits, goes into the prompt on every call, and the model reasons over all of it with no lookup step. Retrieval means an external step (embedding search or another mechanism) picks the slice of a much larger corpus relevant to the current query, and only that slice enters context. Both give the model information beyond its training data. Treat them as two points on a spectrum of how much corpus enters context per call.
2. What a large context window buys
For a corpus that's bounded and known ahead of time (a company's policy documents, a specific codebase, a fixed set of reference materials), putting it in context removes a whole category of failure that RAG failure modes covers: there's no retrieval step to fail. The model has everything. A query that spans several documents, or needs facts scattered across the corpus, doesn't depend on a retriever guessing which few chunks matter, and synthesis questions are exactly what retrieval handles poorly.
The cost is paid on every call. The whole corpus counts against the context window and, without caching, against the cost and latency of every request, whether or not the query needed most of it. This approach is strongest for a corpus that fits in a usable fraction of the window, leaving room for the conversation and the model's reasoning.
3. Where long context stops being the right answer
It breaks down along two independent axes. Size: a corpus larger than the window doesn't fit, and prompt design can't change that. Something must be excluded from each call, which is the selection problem retrieval solves at query time. Volatility: a corpus that changes more often than the cache lifetime has to be re-sent at full cost once the cached version goes stale, which destroys the economics that make a large static corpus in context affordable.
Near the size limit, cost is what tips the decision. Resending a large corpus on every call with no caching benefit, because it changes too often to stay cached, is a compounding cost that a per-query retrieval step avoids by sending only the relevant slice.
4. Where retrieval earns its complexity
Retrieval is justified when the corpus is large or dynamic enough that including all of it in every call is unaffordable and unnecessary, because each query's answer lives in a small fraction of it. A support knowledge base with thousands of articles, most irrelevant to any one question, is the textbook case. Retrieval's per-query cost stays roughly constant as the corpus grows; long context's cost grows with it.
Prompt caching architecture changes this in one specific way. Caching turns a large static portion of a corpus from a per-call cost into a cost paid once and reused, which extends how large a corpus can economically stay in context, provided that portion is stable across requests. It doesn't remove the window's hard size ceiling. It does raise the bar retrieval must clear, for the part of the corpus stable enough to cache.
5. Citations: grounding a claim to a specific span
Neither long context nor retrieval guarantees an answer is traceable to a specific piece of the source; it may be a fluent restatement that happens to agree with it. Citations are a separate capability: the response links each claim to the passage in the provided source that supports it. Without them, the application has to trust that the answer came from the material provided and not from the model's parametric memory of a similar fact, the failure RAG failure modes describes in its section on a model ignoring retrieved context.
This applies however the source arrived, because the risk of a fluent answer with no verifiable anchor exists either way. A cited response gives a downstream consumer, a human reviewer, or an automated check a specific span of a specific document to verify the claim against.
6. A hybrid design: cached long context plus retrieval for the dynamic tail
Most corpora are neither uniformly static nor uniformly dynamic. There's a stable, bounded core (core product documentation, company policies) that rarely changes, plus a much larger, fast-growing tail (individual support tickets, recent product updates) that changes constantly and would never stay cached long enough to be worth including. Put the stable core in context, cached so its cost is paid once, and use retrieval only for the tail, fetching the few recent or specific items each query needs.
Each approach goes where it fits best. Long context serves the core, where synthesis questions benefit from having all of it present and retrieval would do poorly. Retrieval serves the part long context can't economically include.
7. Worked example: a support corpus that outgrew long context
A support application put its whole help-center corpus, a few hundred articles, in a cached system prompt. It worked: queries spanning multiple articles were answered correctly because the model had everything, and caching kept the per-query cost small. Over a year, the corpus grew past the size that fit comfortably alongside the rest of the context budget, and popular articles were updated often enough that the cached copy was frequently stale when a user asked about something that had just changed.
The fix split the corpus. A hand-picked, stable subset (the few foundational articles that rarely change and that most synthesis questions depend on) stayed in context, cached as before. The full corpus, including everything that changes often, moved behind a retrieval step that fetches the articles relevant to each query. Citations come back with each answer, so a support agent can check the response against the current article instead of trusting the model's training or a stale cached copy.
See prompt caching architecture for what makes a large static core affordable to keep in context, and RAG failure modes for debugging the retrieval half once a corpus has outgrown context.