Tokenizer
The component that converts raw text into the sequence of tokens a model actually processes — fixed at pre-training time, and different for every model family.
Before a large language model can understand your text, it has to break it down. That is the job of the tokenizer. It takes raw text and converts it into a sequence of smaller pieces called tokens, and later turns the model's output back into readable words.
Unlike the model itself, a tokenizer is not a neural network. It is a fixed, rules-based system trained before the model even starts its learning process. Most modern models use a method called Byte Pair Encoding to build their vocabulary. This process merges frequent characters into subword units. Common words might become a single token, while uncommon words are split into pieces, and rare words are broken down into individual letters or bytes.
Because of this, different models see the same word in very different ways. Google's Gemini, for instance, uses a massive vocabulary of over two hundred and fifty thousand tokens, while GPT-4 uses a vocabulary of one hundred thousand.
This matters immensely for search engine optimization and branding. If a model breaks your brand name, technical term, or website address into too many fragmented tokens, it becomes harder for the AI to reliably associate those terms with your business. Each fragment carries less training signal than a single, clean token would.
Finally, remember that tokenizers and models are permanently linked. A token ID is just an arbitrary number whose meaning was established during that specific model's training. If you try to mix one model's tokenizer with another model's weights, the output will be complete gibberish. This is why every AI model must always be paired with its own specific tokenizer.
What a tokenizer is
A tokenizer is the component that converts raw text into a sequence of tokens before it enters a language model, and converts the model's output token IDs back into readable text. It is a discrete lookup and splitting system, not a neural network. The tokenizer is trained before the model — on a large vocabulary extracted from the pre-training corpus — and its mapping is then fixed for the lifetime of that model family.
How tokenizers work
Most modern LLM tokenizers use Byte Pair Encoding (BPE) or a variant like SentencePiece or WordPiece. The algorithm starts with individual characters and iteratively merges the most frequent adjacent pairs into single tokens until a target vocabulary size is reached (typically 32,000–200,000 tokens). The result is a vocabulary of subword units: common words are single tokens; uncommon words are split into their component pieces; rare sequences fall back to individual characters or bytes.
Gemini uses SentencePiece with a 256,000-token vocabulary. GPT-4 uses tiktoken (BPE) with a 100,000-token vocabulary. BERT-style models use WordPiece with ~30,000 tokens. Same sentence, three different token sequences.
Why the tokenizer matters for AI SEO
A tokenizer determines how a model "sees" a brand name, a URL, or a technical term. A brand that splits into many unfamiliar subword tokens will be harder for the model to associate reliably with its domain — each component token carries less accumulated training signal than a single-token representation would. DEJAN's tokenizer analysis tool at dejan.ai/tools/tokenizer/ lets you inspect exactly how Gemini tokenises any text and what probability scores each token carries.
Tokenizers and model boundaries
You cannot use one model's tokenizer with another model's weights. The token IDs are arbitrary integers whose meaning was established during that specific model's pre-training. Mixing them produces garbage. This is why every model release includes its tokenizer as a required component alongside the weights.