Watch: Tokenizer
The component that converts raw text into the sequence of tokens a model actually processes — fixed at pre-training time, and different for every model family.
Transcript
Before a large language model can understand your text, it has to break it down. That is the job of the tokenizer. It takes raw text and converts it into a sequence of smaller pieces called tokens, and later turns the model's output back into readable words.
Unlike the model itself, a tokenizer is not a neural network. It is a fixed, rules-based system trained before the model even starts its learning process. Most modern models use a method called Byte Pair Encoding to build their vocabulary. This process merges frequent characters into subword units. Common words might become a single token, while uncommon words are split into pieces, and rare words are broken down into individual letters or bytes.
Because of this, different models see the same word in very different ways. Google's Gemini, for instance, uses a massive vocabulary of over two hundred and fifty thousand tokens, while GPT-4 uses a vocabulary of one hundred thousand.
This matters immensely for search engine optimization and branding. If a model breaks your brand name, technical term, or website address into too many fragmented tokens, it becomes harder for the AI to reliably associate those terms with your business. Each fragment carries less training signal than a single, clean token would.
Finally, remember that tokenizers and models are permanently linked. A token ID is just an arbitrary number whose meaning was established during that specific model's training. If you try to mix one model's tokenizer with another model's weights, the output will be complete gibberish. This is why every AI model must always be paired with its own specific tokenizer.