The average number of bits a source needs per symbol, computed from the probability distribution over its symbols. It sets the floor no lossless compressor can beat, and it is the unit language model quality is reported in.
Shannon entropy measures how much information a source produces per symbol, in bits. Claude Shannon set it out in 1948 as the foundation of information theory. For a source that emits symbol \(i\) with probability \(p_i\), the entropy is
$$H = -\sum_{i} p_i \log_2 p_i$$A source with \(D\) equally likely symbols has entropy \(\log_2 D\), the highest value available to it. A source that always emits the same symbol has entropy zero, because nothing about the next symbol is in doubt. Real text sits between the two: some characters are far more common than others, so each one carries less than the alphabet's capacity would allow.
Entropy is a property of the distribution, not of any single message drawn from it. That distinction is what separates it from Kolmogorov complexity, which measures one specific object rather than the source behind it.
Shannon's source coding theorem states that no lossless code can average fewer than \(H\) bits per symbol. Every compression figure quoted anywhere is a measurement against that floor, which is why compressed size is reported in bits per byte and why the unit carries meaning across formats and languages.
Entropy depends on how much of the sequence you condition on. Order-0 entropy \(H_0\) looks only at how often each symbol appears, ignoring position. Order-1 entropy \(H_1\) is the entropy of a symbol given the one before it, so it registers whether adjacent pairs are predictable. Redundancy expresses the gap between the observed rate and the alphabet's capacity:
$$R = 1 - \frac{H_0}{\log_2 D}$$where \(D\) is the count of distinct symbols. \(R = 0\) is maximal randomness and values toward 1 indicate a skewed, repetitive distribution. Our study of The Verge tracked all three across 152,192 articles and found \(H_0\) falling while \(H_1\) rose, which reads as vocabulary concentrating while local sequential variety increased.
Training a language model minimises cross-entropy: the bits needed to encode the true next token under the distribution the model predicted. The loss function is quoted in bits per token or bits per byte, the same unit as compression, and perplexity is that figure exponentiated, \(2^{H}\).
At inference the same quantity reads as uncertainty. Entropy over the model's next-token distribution at one step is low when a single continuation dominates and high when probability is spread across many candidates. That is the measurement behind token probability work, and the quantity temperature and top-p sampling reshape before a token is drawn. A brand the model is sure of sits in a low-entropy region of that distribution; a brand it is guessing at sits in a high-entropy one.
