The atomic unit a language model reads and generates — a subword chunk, not a whole word — that determines how text is measured, priced, and processed.
A token is the smallest unit of text that a language model processes. It is not necessarily a word: most tokenisation schemes break text into subword pieces, so "tokenization" might become ["token", "ization"], while "cat" stays as a single token. Common words are usually single tokens; rare or technical words may be split into several. Punctuation, spaces, and line breaks are also tokens.
The exact mapping from text to tokens is determined by the model's tokenizer, which was fixed during pre-training. Two models using different tokenizers will split the same sentence differently, which affects how they process and generate it.
Everything in LLM infrastructure is measured in tokens: context window size (how many tokens a model can read at once), API pricing (cost per input and output token), throughput (tokens generated per second), and training data size (how many tokens the model was trained on). A rough rule of thumb is 100 tokens ≈ 75 words in English, but this varies by language and content type — code and non-Latin scripts often tokenise less efficiently.
At each generation step, the model assigns a probability to every possible next token in its vocabulary. The highest-probability tokens are the model's best guesses for what comes next; token probability is also used to diagnose where the model is uncertain about a brand or topic. Temperature and top-p sampling control how the model samples from this distribution.
Brand names, product names, and technical terms may tokenise into multiple subword pieces. A brand name that tokenises cleanly into a single token has a structural advantage: the model represents it as a single unit in its vocabulary, with a direct probability weight. Multi-token names require the model to compose its representation across steps, which can introduce additional uncertainty. Understanding your brand's tokenisation is a minor but real consideration in AI visibility.
