Top-p Sampling
A decoding method that samples the next word from the smallest set of top candidates whose probabilities sum to a threshold p.
When a language model generates text, it has to decide which word comes next. One of the best tools for controlling this choice is top-p sampling, also known as nucleus sampling.
Instead of looking at every possible word, top-p sampling narrows the playing field. It sorts all candidate words by their probability, starts adding them up from the most likely to the least likely, and stops the moment that cumulative probability reaches a set threshold, called "p." The model then chooses the next word only from this select, top-tier group.
You can adjust this threshold to change the style of the output. If you set "p" low, say between point-one and point-five, the model only considers the most highly probable words. This gives you focused, predictable text. If you raise "p" higher, up to point-nine-five, you open the door to a much wider pool of words. This allows for more creative and diverse writing, though it comes with a higher risk of the model drifting off-topic. Setting "p" to one disables the filter entirely, meaning every single word is back on the table.
While other settings like temperature scale the entire range of possibilities, top-p works by simply trimming away the unlikely tail end. They both act on the model's underlying token probabilities, and developers often use them together to get the perfect balance of creativity and control.
Top-p sampling, also called nucleus sampling, controls a model's randomness by limiting which words it will consider next. It sorts every candidate word by probability, adds them up from the top, and keeps only the smallest set whose cumulative probability reaches a threshold p — then samples from that reduced set, weighted by probability.
A low p (say 0.1–0.5) keeps only the most probable words, giving focused, predictable output; a high p (0.75–0.95) admits a wider range, allowing more diverse and creative text at higher risk of drifting off-topic. Setting p to 1 disables the filter entirely, considering all words.
Where temperature rescales the whole distribution, top-p instead trims its tail — the two are often used together. Both act on the model's underlying token probabilities, the same signal our Tree Walker inspects.