The component that converts raw text into the sequence of tokens a model actually processes — fixed at pre-training time, and different for every model family.
A tokenizer is the component that converts raw text into a sequence of tokens before it enters a language model, and converts the model's output token IDs back into readable text. It is a discrete lookup and splitting system, not a neural network. The tokenizer is trained before the model — on a large vocabulary extracted from the pre-training corpus — and its mapping is then fixed for the lifetime of that model family.
Most modern LLM tokenizers use Byte Pair Encoding (BPE) or a variant like SentencePiece or WordPiece. The algorithm starts with individual characters and iteratively merges the most frequent adjacent pairs into single tokens until a target vocabulary size is reached (typically 32,000–200,000 tokens). The result is a vocabulary of subword units: common words are single tokens; uncommon words are split into their component pieces; rare sequences fall back to individual characters or bytes.
Gemini uses SentencePiece with a 256,000-token vocabulary. GPT-4 uses tiktoken (BPE) with a 100,000-token vocabulary. BERT-style models use WordPiece with ~30,000 tokens. Same sentence, three different token sequences.
A tokenizer determines how a model "sees" a brand name, a URL, or a technical term. A brand that splits into many unfamiliar subword tokens will be harder for the model to associate reliably with its domain — each component token carries less accumulated training signal than a single-token representation would. DEJAN's tokenizer analysis tool at dejan.ai/tools/tokenizer/ lets you inspect exactly how Gemini tokenises any text and what probability scores each token carries.
You cannot use one model's tokenizer with another model's weights. The token IDs are arbitrary integers whose meaning was established during that specific model's pre-training. Mixing them produces garbage. This is why every model release includes its tokenizer as a required component alongside the weights.
