Watch: Quadratic Attention Cost
The property that standard transformer attention grows with the square of the input length — doubling the context roughly quadruples the compute and memory — which is why long context windows are expensive and why grounding budgets exist.
Transcript
In a standard AI transformer, every word or token compares itself to every other token. This means if you double the length of your input, you quadruple the computational work and memory required. This is known as quadratic attention cost, and it is the main reason why processing long text is so expensive.
This math shapes the AI products we use every day. It is why longer prompts cost more and take longer to process, and why systems impose strict limits on length. Even when a model theoretically has a massive context window, actually filling it is too costly. For example, AI search engines do not feed entire websites into a model. Instead, they operate on a strict, fixed word budget driven by these economics.
To get around this bottleneck, developers use several clever workarounds. Retrieval-augmented generation keeps contexts short by only pulling in the most relevant passages. Other techniques use mathematical shortcuts and approximations to avoid comparing every single token.
For anyone creating content, this quadratic cost is the driving force behind what AI actually sees. Because these models cannot afford to read everything, they only read a highly curated, budgeted slice. To survive the cut, content must be concise, self-contained, and easy for these algorithms to quickly rank and select.