2.1 Parametric memory and its three problems

A pretrained model’s knowledge lives in its weights. Ask it a question and the answer is reconstructed from patterns fixed at training time — there is no lookup, no source, and no record of where anything came from.

This is called parametric memory (Lewis et al. 2020, sec. 1), and it works remarkably well right up until it doesn’t. A language model’s factual knowledge is stored in its parameters, and models cannot easily expand or revise that memory, explain their predictions, or avoid producing hallucinations (Lewis et al. 2020, sec. 1).

Those three failures are worth separating, because they have different consequences:

It goes stale. Knowledge is frozen at the training cut-off. Anything that happened since, or any document the model never saw — your company’s internal wiki, a contract signed last week — is simply absent.

It cannot cite. The model produces an answer, not a provenance trail. You cannot check the claim against a source, because there is no source; there is a distribution over tokens.

It fails silently. A model asked about something outside its knowledge does not return an error. It produces the most plausible continuation, which is exactly what a confident wrong answer looks like.

Retraining fixes none of this well. It is expensive, it has to be repeated, and it still leaves you unable to say where an answer came from.

3.1 Dense retrieval

The alternative is to stop asking the model to remember, and start giving it the relevant text at question time. That turns the problem into search — and search over meaning rather than words.

Keyword search fails here in an obvious way: a question phrased differently from the document that answers it will not match. The fix is to represent both as vectors positioned by meaning, so that closeness in the vector space corresponds to closeness in sense.

3.2 Two encoders and a dot product

Dense passage retrieval encodes questions and passages with separate encoders and scores them by the dot product of the two embeddings (Karpukhin et al. 2020, sec. 3.1).

Separate encoders matter more than it first appears. Because passages are encoded independently of any question (Karpukhin et al. 2020, sec. 3.1), every passage in the corpus can be embedded once, in advance. At query time only the question needs encoding — after which finding matches is arithmetic over vectors rather than running a model against millions of documents.

The approach beats keyword matching by a wide margin. Dense retrieval outperformed a strong Lucene-BM25 baseline by 9 to 19 percentage points absolute in top-20 passage retrieval accuracy (Karpukhin et al. 2020, abstract). On Natural Questions, dense passage retrieval reached 78.4% top-20 accuracy against 59.1% for BM25 (Karpukhin et al. 2020, table 2).

3.3 Making it fast

Comparing a query against millions of vectors exactly is too slow to sit in front of a user, so production systems approximate. Approximate nearest-neighbour indexes such as FAISS make similarity search tractable over millions to billions of vectors (Karpukhin et al. 2020, sec. 3.1).

The trade is explicit: you accept a small chance of missing the true nearest neighbour in exchange for search that returns in milliseconds. For retrieval feeding a language model, that is a good trade — the model is going to read several candidates anyway.

3.4 Putting it together, and when not to

Put the two halves together and you get the architecture the field settled on. Retrieval-augmented generation pairs a parametric memory, a pretrained seq2seq model, with a non-parametric memory, a dense vector index of passages accessed by a retriever (Lewis et al. 2020, sec. 2).

The division of labour is the point. The weights hold language — grammar, reasoning patterns, how to write a coherent answer. The index holds facts. And facts, unlike language, change.

That yields a property worth dwelling on: a retrieval system’s world knowledge can be updated by replacing its non-parametric index, without retraining the model (Lewis et al. 2020, sec. 4.5). Swapping an index is minutes of work. Retraining is not.

3.5 What it does and does not fix

The RAG authors report qualitatively that their models hallucinate less and produce factually correct text more often than a comparable parametric-only baseline; the measured factuality comparison is the separate human evaluation (Lewis et al. 2020, sec. 4.2; 2020, sec. 4.3). In that evaluation of Jeopardy question generation, RAG output was judged more factual in 42.7% of cases against 7.1% for the BART baseline (Lewis et al. 2020, sec. 4.3).

Read those numbers carefully, because they are routinely overstated. A six-fold improvement in judged factuality is a large effect. It is not elimination, and “42.7% of cases judged more factual” is not “42.7% accurate” — it is a comparison between two systems, on one task, judged by people.

So retrieval solves the three problems from earlier: staleness, because the index can be replaced; provenance, because the retrieved passage is a citable artifact; and coverage, because private documents can be indexed without touching the model.

It does not solve reasoning. A model handed exactly the right passage can still misread it, over-generalise from it, or blend it with something half-remembered from pretraining. Retrieval changes what the model can see. It does not change how well it thinks about what it sees — and treating a retrieval system as a correctness guarantee is the most common way teams get burned.