Watch: Tokenization
Breaking text into subword tokens a model can process; each token carries a frequency bias from how often it appeared in training.
Transcript
Before an artificial intelligence model can understand your text, it has to break it down into smaller pieces called tokens. Common words usually stay whole, while rare or complex words get chopped up into smaller fragments. This process, known as tokenization, is how the model keeps its vocabulary manageable while still processing unfamiliar terms.
But there is a hidden bias built right into this system. If we look closely at a tokenizer, like the one used by Google's Gemini, we find that every single token carries a learned score based on how often it appeared in the model's training data. Frequent words and fragments score much higher.
This bias flows directly downstream, influencing how the model interprets your prompts and queries. For search engine optimization and AI content creation, this means your choice of words has a hidden cost. Rare, highly fragmented terms can register as unfamiliar and uncertain to the AI. Ultimately, tokenization is the invisible foundation that shapes how models perceive word rarity and decide which ideas are familiar, and which are not.