Watch: BlockRank

Large language models are incredibly powerful at ranking search results, but they are usually too slow and expensive to use on big pools of documents. Standard in-context ranking requires the model to compare everything to everything else, creating a massive computational bottleneck.

Researchers from the University of Texas at Austin and Google have found a way to break this bottleneck with a method called BlockRank. They noticed two key things about how a fine-tuned ranking model actually pays attention. First, attention is very dense within a single document, but sparse across different documents. Second, specific parts of the model—called retrieval heads—reliably point directly to the most relevant documents in the middle layers of the network.

BlockRank exploits this by restructuring how the model attends to information. Instead of letting every word look at every other word, document text is restricted to attending only to itself and the shared instructions. The query, however, can still attend to everything. By adding a contrastive training objective, BlockRank turns this query-to-document attention into a highly dependable signal for relevance.

The results are dramatic. BlockRank can rank up to five hundred documents in less than a second. That is nearly five times faster than a standard fine-tuned Mistral model, all while matching or beating the state of the art in accuracy. For search and relevance engineering, it is a major step toward making high-quality generative retrieval affordable at web scale.