Listen: Shannon Entropy
The average number of bits a source needs per symbol, computed from the probability distribution over its symbols. It sets the floor no lossless compressor can beat, and it is the unit language model quality is reported in.
Transcript
In 1948, Claude Shannon introduced a concept that became the foundation of information theory: Shannon entropy. Essentially, it measures how much information a source produces, calculated in bits.
Think of it as a measure of surprise. If a source always spits out the exact same character, there is no surprise, and the entropy is zero. If every possible character is equally likely, surprise is at its peak, and entropy is at its maximum. Real-world language sits somewhere in the middle, because some letters and words are much more common than others.
Shannon proved that entropy sets an absolute physical limit. No lossless compression algorithm can ever compress data to be smaller than its entropy floor. This is the ultimate limit for every compressed file we use, regardless of format or language.
The exact value of entropy changes depending on how much context we look at. Order-zero entropy only counts how often individual symbols appear, ignoring their position. Order-one entropy looks at pairs, predicting the next symbol based on the one right before it.
Today, this same math powers language models. Training an AI is all about minimizing cross-entropy, which is the number of bits needed to predict the next token. When a model is highly confident about what comes next, its entropy is low. When it is guessing among many possibilities, its entropy is high. This uncertainty is the exact distribution we reshape when we adjust settings like temperature and top-p sampling.