BlockRank
A method that makes LLM in-context ranking scale to large candidate sets by exploiting the block-sparse structure of attention.
Large language models are incredibly powerful at ranking search results, but they are usually too slow and expensive to use on big pools of documents. Standard in-context ranking requires the model to compare everything to everything else, creating a massive computational bottleneck.
Researchers from the University of Texas at Austin and Google have found a way to break this bottleneck with a method called BlockRank. They noticed two key things about how a fine-tuned ranking model actually pays attention. First, attention is very dense within a single document, but sparse across different documents. Second, specific parts of the model—called retrieval heads—reliably point directly to the most relevant documents in the middle layers of the network.
BlockRank exploits this by restructuring how the model attends to information. Instead of letting every word look at every other word, document text is restricted to attending only to itself and the shared instructions. The query, however, can still attend to everything. By adding a contrastive training objective, BlockRank turns this query-to-document attention into a highly dependable signal for relevance.
The results are dramatic. BlockRank can rank up to five hundred documents in less than a second. That is nearly five times faster than a standard fine-tuned Mistral model, all while matching or beating the state of the art in accuracy. For search and relevance engineering, it is a major step toward making high-quality generative retrieval affordable at web scale.
BlockRank is a method from UT Austin and Google that makes in-context ranking with LLMs fast enough for large candidate sets. Standard in-context ranking is quadratic in cost; BlockRank cuts that to roughly linear by restructuring how the model attends.
It rests on two observations about a fine-tuned ranking model's attention: attention is dense within a document but sparse across documents, and certain end-of-query "retrieval heads" reliably point at the relevant document in the model's middle layers. BlockRank enforces that structure — document tokens attend only within their own document plus shared instructions, while query tokens attend to everything — and adds a contrastive training objective so query-to-document attention becomes a dependable relevance signal.
The payoff is scale: it can rank up to 500 documents in under a second, runs about 4.7× faster than a fine-tuned Mistral-7B baseline, and matches or beats prior state of the art. For relevance engineering it's a glimpse of how generative retrieval stays affordable at web scale.