A method that makes LLM in-context ranking scale to large candidate sets by exploiting the block-sparse structure of attention.
BlockRank is a method from UT Austin and Google that makes in-context ranking with LLMs fast enough for large candidate sets. Standard in-context ranking is quadratic in cost; BlockRank cuts that to roughly linear by restructuring how the model attends.
It rests on two observations about a fine-tuned ranking model's attention: attention is dense within a document but sparse across documents, and certain end-of-query "retrieval heads" reliably point at the relevant document in the model's middle layers. BlockRank enforces that structure — document tokens attend only within their own document plus shared instructions, while query tokens attend to everything — and adds a contrastive training objective so query-to-document attention becomes a dependable relevance signal.
The payoff is scale: it can rank up to 500 documents in under a second, runs about 4.7× faster than a fine-tuned Mistral-7B baseline, and matches or beats prior state of the art. For relevance engineering it's a glimpse of how generative retrieval stays affordable at web scale.
Paper: Scalable In-context Ranking with Generative Models
