Token
The atomic unit a language model reads and generates — a subword chunk, not a whole word — that determines how text is measured, priced, and processed.
In the world of large language models, the fundamental unit of measurement is the token. A token is the smallest piece of text a model can process. While common words like "cat" might be a single token, longer or less common words are often broken down into subword pieces. For example, the word "tokenization" might be split into "token" and "ization." Punctuation, spaces, and line breaks count as tokens too. As a rough rule of thumb, one hundred tokens is about seventy-five words in English, though non-Latin scripts and computer code are usually less efficient.
Tokens matter because they define how artificial intelligence infrastructure operates. Everything from API pricing and processing speed to the size of a model's memory is measured in tokens.
When generating text, a model works by predicting the very next token, assigning a probability to every possible choice in its vocabulary. This has surprising implications for search engine optimization and brand visibility. If a brand or product name cleanly fits into a single token, the model can represent it as one direct unit. If a name is split into multiple tokens, the model has to build that representation step-by-step, which can introduce uncertainty. Understanding how your brand translates into tokens is a small but highly valuable detail in the age of AI.
What a token is
A token is the smallest unit of text that a language model processes. It is not necessarily a word: most tokenisation schemes break text into subword pieces, so "tokenization" might become ["token", "ization"], while "cat" stays as a single token. Common words are usually single tokens; rare or technical words may be split into several. Punctuation, spaces, and line breaks are also tokens.
The exact mapping from text to tokens is determined by the model's tokenizer, which was fixed during pre-training. Two models using different tokenizers will split the same sentence differently, which affects how they process and generate it.
Why tokens are the unit of measurement
Everything in LLM infrastructure is measured in tokens: context window size (how many tokens a model can read at once), API pricing (cost per input and output token), throughput (tokens generated per second), and training data size (how many tokens the model was trained on). A rough rule of thumb is 100 tokens ≈ 75 words in English, but this varies by language and content type — code and non-Latin scripts often tokenise less efficiently.
Token probability
At each generation step, the model assigns a probability to every possible next token in its vocabulary. The highest-probability tokens are the model's best guesses for what comes next; token probability is also used to diagnose where the model is uncertain about a brand or topic. Temperature and top-p sampling control how the model samples from this distribution.
AI SEO relevance
Brand names, product names, and technical terms may tokenise into multiple subword pieces. A brand name that tokenises cleanly into a single token has a structural advantage: the model represents it as a single unit in its vocabulary, with a direct probability weight. Multi-token names require the model to compose its representation across steps, which can introduce additional uncertainty. Understanding your brand's tokenisation is a minor but real consideration in AI visibility.