Attention
The transformer mechanism that lets a model weigh how much every token relates to every other, in parallel — the core of modern LLMs.
Attention is the mechanism that lets a language model weigh how much each token relates to every other token when building meaning. Introduced in the 2017 paper "Attention Is All You Need," it powers the Transformer, the architecture behind essentially every modern large language model.
Its breakthrough was removing the need for step-by-step recurrence: attention relates all positions in a sequence simultaneously, which makes training highly parallel and lets the model connect distant words directly. Refinements like scaled dot-product attention (dividing by √dₖ to keep gradients stable) and multi-head attention (several attention "views" at once, each focusing on different relationships) make it both stable and expressive.
Attention's one weakness is cost — it scales quadratically with input length, which is exactly why methods like BlockRank restructure it for large-scale in-context ranking. Understanding attention is the key to understanding how models actually read.