Breaking text into subword tokens a model can process; each token carries a frequency bias from how often it appeared in training.
Tokenization is the step where a model breaks text into smaller pieces — subword "tokens" — before it can process it. Common words stay whole while rare words split into fragments (for example "quantumization" → "quant", "um", "ization"), which keeps the vocabulary manageable while still handling unseen words.
We dissected Gemini's actual SentencePiece tokenizer to show that tokens carry an inherent bias: each token has a learned log-likelihood score reflecting how prominent that subword was in training. Frequent subwords score higher, and that bias flows downstream into how the model treats your content, prompts and queries.
For AI SEO this means word choice has a hidden cost: rare, fragmented terms can register as less familiar to the model. Tokenization is the foundation beneath token probability and everything our Tree Walker reads about word rarity and uncertainty.
