Watch: Dissecting Gemini’s Tokenizer and Token Scores
Explore how Google’s Gemini processes text using subword tokenization. Use this tool to inspect SentencePiece log-likelihood scores for common and rare tokens.
Transcript
Search engine optimization professionals diving into machine learning often need to understand how models like Google’s Gemini process text. It all starts with subword tokenization, which breaks words down into smaller pieces called tokens. This approach strikes a balance. Common words stay whole, while rare words are split into smaller, recognizable pieces so the model can still understand them.
Gemini uses a system called SentencePiece, which features a vocabulary of two hundred and fifty-six thousand distinct tokens. During training, every single token is assigned a score. This score represents how essential that piece was for reconstructing the training data. It is a global, context-independent measure of how common a token is.
For example, extremely common English subwords, like the word the or the suffix i n g, receive very high scores. On the other hand, obscure symbols, rare emojis, and special control tokens sit at the very bottom of the scale.
When you analyze these scores, you are seeing the underlying building blocks of the language model. Understanding which tokens are common and which are rare helps you see exactly how Gemini interprets and processes the text you throw at it.