Watch: CHISEL
Chunk Holdout Influence Scoring by Elimination Loss: a Shapley-value method that measures how many rank places each chunk of an AI-cited source's text is worth.
Transcript
When an AI citation ranks a source, how do we know which specific parts of the text actually drove that ranking? A process called CHISEL, or Chunk Holdout Influence Scoring by Elimination Loss, helps find the answer.
It starts by taking a cited source and cutting its text into smaller chunks, like sentences or paragraphs. If there are too many chunks, they are grouped into topic blocks. Next, a language model ranks the sources over and over again. Each time, it looks at a different combination of these chunks. By systematically leaving chunks out and putting them back in, the system calculates a score for each one.
This score represents the average number of rank places a source gains when a specific chunk is present. A positive score means the text helps the ranking, while a negative score means it actually pulls the ranking down.
Simply removing one chunk at a time doesn't work. In testing, sixty percent of single-chunk removals showed no change in rank at all, completely missing the true impact of the text. By testing multiple combinations instead, CHISEL reveals the hidden value of every sentence.