How well a model predicts text: the exponential of average per-token cross-entropy, tied to entropy, and the standard intrinsic metric for evaluating language models. Not the search company of the same name.
Perplexity is a measure of how well a language model predicts a piece of text: the lower the perplexity, the less surprised the model is by what comes next. It ties directly to entropy: for a per-token entropy \(H\) measured in bits, perplexity is \(2^{H}\), the effective number of equally likely choices the model is deciding between at each step.
It is the standard intrinsic metric for evaluating and comparing language models and for tracking one during pre-training: a model that has learned the patterns of a language assigns high probability to real text and so scores low perplexity. It measures fluency and fit, not truth, so a confident, low-perplexity output can still be a hallucination.
Perplexity is the flip side of information density: text that is highly predictable carries little new information, while text that is dense and specific is less predictable. Do not confuse the metric with the AI search product of the same name.
