Listen: Token
The atomic unit a language model reads and generates — a subword chunk, not a whole word — that determines how text is measured, priced, and processed.
Transcript
In the world of large language models, the fundamental unit of measurement is the token. A token is the smallest piece of text a model can process. While common words like "cat" might be a single token, longer or less common words are often broken down into subword pieces. For example, the word "tokenization" might be split into "token" and "ization." Punctuation, spaces, and line breaks count as tokens too. As a rough rule of thumb, one hundred tokens is about seventy-five words in English, though non-Latin scripts and computer code are usually less efficient.
Tokens matter because they define how artificial intelligence infrastructure operates. Everything from API pricing and processing speed to the size of a model's memory is measured in tokens.
When generating text, a model works by predicting the very next token, assigning a probability to every possible choice in its vocabulary. This has surprising implications for search engine optimization and brand visibility. If a brand or product name cleanly fits into a single token, the model can represent it as one direct unit. If a name is split into multiple tokens, the model has to build that representation step-by-step, which can introduce uncertainty. Understanding how your brand translates into tokens is a small but highly valuable detail in the age of AI.