RAG versus long context

The second approach skips retrieval: make the context window large enough and pass the entire file instead of chunks. Today’s models have large windows, so this is simple. No chunking, no embedding, no vector DB, no reranking, no sync strategy.

For a small corpus it is the right call. When Anthropic introduced contextual retrieval, its advice was that a knowledge base under 200,000 tokens, about 500 pages, can go straight into the prompt.

Where it hurts

The whole document is re-read on every request, and cost grows with it. Prompt caching softens this, but a cache covers a prompt prefix. It pays when the same document leads each request and only the question changes. Content that differs per query is read afresh.

Attention dilutes as the context grows, so the model can miss what is in it. Research on long contexts found performance highest when the relevant text is at the beginning or end, and worse in the middle (see rerankingContext rerankingModels read the middle of a long context worst, so put what matters at the edges.Opens in a new tab). RAG filters first and hands the model a smaller, focused set.

And the window is still a limit. A corpus past it cannot be passed whole at all.

Side by side

Concern	RAG	Long context
Moving parts	Chunking, embedding, vector DB, reranking, sync	None, the file is the context
Cost per query	Reads only the retrieved chunks	Re-reads the whole document, caching helps a repeated prefix
Attention	Filtered, focused context	Diluted, the middle can be missed
Failure mode	Retrieval misses, the model sees partial data	The window caps the corpus size

Neither removes the limit. RAG still has to fit its results in the window, and long context needs the corpus to fit. Anthropic’s context engineering guidance points to a hybrid for agents: retrieve some data up front for speed, and let the agent explore for the rest. See context engineeringContext engineeringDeciding what the model sees on every call, because the window is finite.Opens in a new tab and overflow managementContext overflowWhen the history outgrows the window: cut the oldest, or summarize them.Opens in a new tab.