Retrieval-Augmented Generation
Delegate the Finding
Asking a model to read your entire repository is like asking a senior architect to photocopy the plans. It works, but it burns the judgment you hired.
So delegate the looking. You have already met this move as sub-agent delegation: the finding happens in another window, and only a synthesis comes back. Finding is two jobs, and neither needs a frontier model:
- Agentic search — fan out and follow the trail: web search, repository search, or a walk across a linked graph, until the evidence is in hand.1
- Summarization — compress what came back into the few facts that matter, dropping the rest.2
Code and knowledge bases look like plain lists of text, but they are graphs wearing a flat costume. Modules import modules, functions call functions, types inherit types: the repository is a network, and finding your way through it is a traversal. A knowledge base is usually the same story — one fact wired to the next. The GraphWalk benchmark is built to probe exactly that walk: whether a model can move through a graph step by step, following the edges, wherever the structure leads.3
Both are standard in agentic models, so a small one can run the loop and hand the frontier model a short, distilled brief.4 You pay the cheap model for breadth and the frontier model for judgment. The payoff is not the search; it is everything the frontier model never has to do: scan the corpus, rank candidates, hold the noise. It never reads the library; it reads the page you hand it.
RAG: The Same Delegation, Scaled
Delegation is the opening move. Retrieval-augmented generation (RAG) is that same move, scaled. A language model already contains the layers that turn text into numbers; strip away everything else and what remains is a small encoder you can run on its own. Marry that encoder to the database's old trick of pre-indexing, and you get a system that finds the relevant pieces of information across a very large knowledge base without reading all of it. Delegation scales by trading a less capable worker for work done ahead of time.
Embeddings: Meaning Becomes Distance
To search by meaning, meaning has to become a number. An embedding model maps text to a dense vector — a coordinate where distance is relatedness. Related text lands close: cat scores 0.91 against feline and 0.06 against rocket. Meaning becomes geometry, and relatedness becomes arithmetic.
Finding related things is now a proximity problem — one the field solved years ago, with the same machinery behind "find similar images," the "you might also like" row, and grouping photos by face. Retrieval borrows it: once items are coordinates, closeness is a lookup.
Reliability levers
Meaning becomes distance, then gets indexed
The model doing this is small enough not to be the bottleneck, and cheap enough not to show up on the bill. Embedding is a commodity: Google's EmbeddingGemma fits a competitive model in under 200MB of RAM, Qwen's Qwen3-Embedding-8B ships as open weights you can run for free, and Voyage's hosted voyage-4-lite bills $0.02 per million tokens. Generation is a different economy: a frontier model like Claude Opus 5 charges $5 per million tokens to read the same text and $25 to write it. Embedding the whole corpus is a rounding error, and you pay it once.5
That price is possible because questions and documents are encoded independently by the same model, so document vectors can be computed ahead of time.6 Indexing is the rest of the setup: slice the corpus into passages, encode each one, and keep the results in a structure made for quick lookup — in practice an HNSW graph.7
Semantic Search: Finding the Nearest Few
Now the read side — the payoff of the scaled delegation. The costly preparation is behind you, so asking costs almost nothing. At query time you embed the question with the same model that built the index — vectors from different models live in different spaces, so mixing them is meaningless. The question is one context, so it becomes one vector, and that is the only new embedding the whole query costs. Then you ask the index one question: which stored vectors sit nearest to this one? That is a similarity search, and geometrically it is just finding the query vector's nearest neighbours. The chunks behind those neighbours are the candidates.
This is what "search by meaning" buys: doctor pay retrieves physician salary even though the two share no words.
At toy scale the index can compare the query against every vector and take the closest. At corpus scale that linear scan is too slow, so the index is built for approximate nearest-neighbour (ANN) search: it returns the nearest few it can find quickly rather than the exact nearest few. That is the trade being made — recall for latency — and the recall/latency dial is most of what "approximate top-k" means in practice.
Reliability levers
One question in, the nearest few passages out
The economics are the ones a database taught long ago: build the organizing structure up front, then each read is a short hop instead of a full sweep. That is what earns retrieval its place — a small model fetches a handful of passages,8 and the reader spends attention on those rather than on the entire library. Weighted against pouring everything into a long context, this is the leaner, more focused option; it is not automatically the truer one.9
Deep Research: The Same Delegation, Iterated
Answering a narrow question against what you already prepared is one hop. Deep research takes on questions whose clues are scattered across sources nobody has gathered, where the path is not known in advance. Strip away the branding and the shape is constant: exploration gathers the candidate pieces of reality, and synthesis turns them into an answer.
Exploration gathers candidate reality. It runs on two retrieval arms: semantic search for meaning, and plain literal search — keyword and structural lookups — for the exact identifier, the rare name, and the thing whose meaning you cannot paraphrase. The two miss different things, so real systems run both. An LLM usually drives the loop: it reads what came back, reasons about what is still unanswered, and issues the next query. So exploration is rarely one shot — each fact reveals the next question, and retrieval and reasoning alternate instead of firing one fixed query. On multi-hop questions that alternation beats any single query.10
Synthesis decides what earns context. Its binding constraint is attention, not window size: past a point more evidence lowers quality instead of raising it (the attention curve).1112 The work is choosing: rerank, dedupe, order, and compress so the strongest evidence sits where the model actually reads, summarizing in parallel and merging the summaries when it still overflows one pass. Provenance has to survive the journey — every claim should trace to the passage that supports it, because even the best systems fail to fully support their citations about half the time.13
Reliability levers
Two phases: explore, then synthesize
Most chat interfaces already ship it: flip on deep research, ask the sprawling question, and wait. What comes back is written, cited, and compact. Underneath, it is the grounding phase run as a workflow agent: code owns the loop and the stop condition, the model is called to judge what is missing and to write. The orchestrator receives the conclusion; the raw pile never reaches it.
Those products aim it at the web, where the retrieval tools already exist. The technique is the part that travels: the same loop runs over a codebase, a ticket archive, or a shelf of documents, reading symbols and line ranges, or sections and links, instead of search results. The reading changes; the loop does not. Length bends the same way — the same run sizes its evidence to the page, so it returns a paragraph or a long report. ChunkHound — a sister project of this book — runs that same loop over code, git history, and the web, locally on the developer's laptop, so nothing leaves the machine.
A run is a different order of cost: minutes rather than milliseconds, and hundreds of documents rather than a handful.14 When a question is self-contained, an ordinary lookup settles it in one round trip — deep research is for the answer you cannot reach in one step.
Dedicate an Agent to Retrieval
These aren't alternatives you pick once — they're stages, and a workflow climbs them as it grows. Each stage answers the limit the one before it hit, and the direction never changes: more of the finding happens before the thinking window, and less guesswork reaches it.
The strongest form wraps all three in one grounding agent: the same delegation, aimed at a single task instead of a general one. It is a workflow agent whose only job is retrieval for this task. It knows what is being asked, runs both arms, and delivers a compact, cited selection to the main agent, so the search is never billed to the thinking. A generic retriever guesses at relevance; a task-scoped one is aimed at it.
What decides which stage you're on is how far the work has grown. An agent that outgrows a single window delegates the finding — the searching moves out of the window, the thinking stays. When the material outgrows one pass, the next stage precomputes the search: vectors stored once, every question answered by proximity. When the question itself escapes what you prepared, the whole arrangement gives way to a dedicated deep research workflow agent, one whose only job is to ground the main agent that does the actual task. Grounding is the constant underneath: every claim traces to a source actually retrieved.