Chunk Holdout Influence Scoring by Elimination Loss: a Shapley-value method that measures how many rank places each chunk of an AI-cited source's text is worth.
CHISEL (Chunk Holdout Influence Scoring by Elimination Loss) is the process behind grounding snippet ablation. It measures how many rank places each chunk of a cited source's text is worth, by holding chunks out across many combinations and averaging the rank the source loses when each one is eliminated.
The process:
- Run a prompt on an AI platform and take the sources its answer cited, with the text the platform read from each. The source cited first is the target.
- Cut the target's text into chunks (sentences, paragraphs or the extracts between "..." separators) that join back into the original text exactly. Every mention of the target's brand counts as one more chunk. When there would be more than 30 chunks, consecutive chunks are grouped into topic blocks so that no more than 30 remain.
- Rank the cited sources with a language model, many times, each time with a different subset of the target's chunks present. Chunks are added one at a time in random orders, and each order is also walked in reverse.
- Score each chunk by its Shapley value: the average number of rank places the target gains when the chunk is present. The values add up to the difference between the target's rank with no text and with its full text.
- Stop when every value is known to within 0.25 rank places, when the budget is spent, or when every combination has been ranked.
A positive value means the chunk holds the source up, and a negative value means it pulls the source down. Removing one chunk at a time misses most of these effects: in 111 runs, 60% of single-chunk removals left the rank unchanged, including lines with clearly non-zero values.
Paper: CHISEL: Chunk Holdout Influence Scoring by Elimination Loss (PDF). The method runs in the DEJAN app under Content Optimizer → Ablation.
Related concepts
