Watch: Perplexity
How well a model predicts text: the exponential of average per-token cross-entropy, tied to entropy, and the standard intrinsic metric for evaluating language models. Not the search company of the same name.
Transcript
In the world of artificial intelligence, perplexity is a key metric used to judge how well a language model predicts text. Put simply, the lower the perplexity, the less surprised the model is by what comes next.
It is directly tied to the concept of entropy. In simple terms, perplexity represents the effective number of equally likely words the model is deciding between at each step.
This makes it the standard tool for evaluating language models during training. A model that has successfully learned the patterns of a language will assign a high probability to real-world text, resulting in a low perplexity score.
But there is a catch. Perplexity measures fluency and grammatical fit, not truth. This means a model can confidently output a complete hallucination while still maintaining a very low perplexity.
Ultimately, perplexity is the flip side of information density. Highly predictable text carries very little new information, while dense, specific writing is much harder to predict. And just to be clear, this mathematical metric is not the same thing as the AI search engine that shares its name.