Watch: Top-p Sampling
A decoding method that samples the next word from the smallest set of top candidates whose probabilities sum to a threshold p.
Transcript
When a language model generates text, it has to decide which word comes next. One of the best tools for controlling this choice is top-p sampling, also known as nucleus sampling.
Instead of looking at every possible word, top-p sampling narrows the playing field. It sorts all candidate words by their probability, starts adding them up from the most likely to the least likely, and stops the moment that cumulative probability reaches a set threshold, called "p." The model then chooses the next word only from this select, top-tier group.
You can adjust this threshold to change the style of the output. If you set "p" low, say between point-one and point-five, the model only considers the most highly probable words. This gives you focused, predictable text. If you raise "p" higher, up to point-nine-five, you open the door to a much wider pool of words. This allows for more creative and diverse writing, though it comes with a higher risk of the model drifting off-topic. Setting "p" to one disables the filter entirely, meaning every single word is back on the table.
While other settings like temperature scale the entire range of possibilities, top-p works by simply trimming away the unlikely tail end. They both act on the model's underlying token probabilities, and developers often use them together to get the perfect balance of creativity and control.