Tokenization
Breaking text into subword tokens a model can process; each token carries a frequency bias from how often it appeared in training.
Before an artificial intelligence model can understand your text, it has to break it down into smaller pieces called tokens. Common words usually stay whole, while rare or complex words get chopped up into smaller fragments. This process, known as tokenization, is how the model keeps its vocabulary manageable while still processing unfamiliar terms.
But there is a hidden bias built right into this system. If we look closely at a tokenizer, like the one used by Google's Gemini, we find that every single token carries a learned score based on how often it appeared in the model's training data. Frequent words and fragments score much higher.
This bias flows directly downstream, influencing how the model interprets your prompts and queries. For search engine optimization and AI content creation, this means your choice of words has a hidden cost. Rare, highly fragmented terms can register as unfamiliar and uncertain to the AI. Ultimately, tokenization is the invisible foundation that shapes how models perceive word rarity and decide which ideas are familiar, and which are not.
Tokenization is the step where a model breaks text into smaller pieces — subword "tokens" — before it can process it. Common words stay whole while rare words split into fragments (for example "quantumization" → "quant", "um", "ization"), which keeps the vocabulary manageable while still handling unseen words.
We dissected Gemini's actual SentencePiece tokenizer to show that tokens carry an inherent bias: each token has a learned log-likelihood score reflecting how prominent that subword was in training. Frequent subwords score higher, and that bias flows downstream into how the model treats your content, prompts and queries.
For AI SEO this means word choice has a hidden cost: rare, fragmented terms can register as less familiar to the model. Tokenization is the foundation beneath token probability and everything our Tree Walker reads about word rarity and uncertainty.