Watch: In-Context Ranking
Re-ranking candidate documents for a query by feeding them all into an LLM's context and letting it pick the most relevant.
Transcript
In-context ranking is a powerful new way generative AI systems decide which information is most relevant. Instead of relying on traditional keyword scores, a large language model is given a query, a list of candidate documents, and a task description all at once. The model reads everything in its context window and decides on the best order. This is a core part of modern relevance engineering, and it is reshaping how search engine optimization works for AI.
The major drawback of this approach is the cost. Because the attention mechanism in language models scales quadratically, doubling the number of documents can quadruple the required computing power. This bottleneck makes ranking large lists of candidates highly impractical.
To solve this problem, a new technique called BlockRank is being used to exploit the block-sparse structure of attention. This allows systems to leverage the deep contextual understanding of language models for ranking, without the massive computational bottleneck.