Listen: Context Window
The maximum number of tokens a model can read at once — the total working memory available for a prompt, grounding documents, conversation history, and the generated response combined.
Transcript
An artificial intelligence model's context window is its memory limit for a single conversation. Everything the model needs to understand—including your prompt, past messages, and any documents you upload—must fit inside this window. If information falls outside of it, the model simply cannot see it.
The size of this window is crucial for grounding, which is how a model uses real-world data to answer questions. While older models were limited to just a few thousand tokens, newer ones have massive capacities. For example, OpenAI's GPT-4o can handle one hundred and twenty-eight thousand tokens, while Google's Gemini 1.5 Pro can process a staggering one million tokens. This is enough to hold roughly three-quarters of a million words, or hundreds of web pages.
However, a massive context window doesn't mean a model reads everything perfectly. Research shows that language models suffer from a "lost-in-the-middle" problem. They pay close attention to the very beginning and the very end of a long document, but often overlook details buried in the middle.
Furthermore, real-world constraints like latency and cost mean search engines often use a fixed word budget for grounding, regardless of how large the model's theoretical window is. For anyone looking to optimize content for AI search, this means the position of your information matters just as much as its presence. To be remembered, your key points and brand mentions need to be right at the top or at the very end.