Parametric Memory

Parametric Memory is the knowledge a model holds in its own weights, learned during pre-training and recalled in a single forward pass with no lookup and no retrieval. Lewis et al. named it in the 2020 RAG paper to separate it from non-parametric memory, the external index a system searches at request time and loads into the context window.

Where the knowledge sits

Facts are spread across the weights rather than held in addressable records. Geva et al. (2021) showed that the feed-forward layers of a transformer work as key-value memories: each key pattern responds to a class of inputs, and the paired value shifts the output distribution toward particular tokens. Mechanistic interpretability work such as ROME (Meng et al., 2022) traces a single factual association to a small set of middle-layer MLP weights and rewrites it there, which shows the storage is localised enough to change one fact at a time.

How much fits

Allen-Zhu and Li (2024) measured knowledge capacity at about 2 bits per parameter when each fact appears around 1,000 times in training, falling to about 1 bit per parameter at 100 exposures. At the 2-bit figure a 7-billion-parameter model holds roughly 14 billion bits, near 1.75 GB, of factual content. Exposure count is the binding constraint: a fact seen a few times in the training data is encoded weakly or not at all, while one repeated across many sources is recalled reliably.

Frozen at the cutoff

Parametric memory is fixed when training stops, which is what the knowledge cutoff describes. Changing it takes fine-tuning, a further pre-training run, or targeted weight editing, and none of those run per query. Anything later reaches the model only through grounding or retrieval-augmented generation, which supply text at inference and leave the weights untouched.

Failure mode

A query that lands on a weakly encoded fact still returns fluent text, because generation samples the most probable continuation whether or not the knowledge is present. That is the mechanism behind hallucination: the weights carry no separate signal for "this was never stored".

Parametric memory and AI visibility

A brand that appears widely and consistently across the training corpus is answered from parametric memory, with no retrieval step and no citation attached. A brand that does not exists for the model only when it can be retrieved, which puts the weight on crawlability and freshness. The two paths produce different results in an AI answer: parametric recall returns a claim with no link back to a source, while retrieved content can be cited.

Related concepts

• Pre-training
• Training Data
• Knowledge Cutoff
• Retrieval-Augmented Generation
• Grounding
• Hallucination
• Fine-tuning
• Mechanistic Interpretability