Measuring how many rank places each part of an AI-cited source's text is worth, by ranking the cited sources many times with different parts removed.
Grounding snippet ablation measures how many rank places each part of a cited source's text is worth. The sources an AI answer cited are ranked many times, each time with a different combination of the target source's spans removed. Each span then gets its Shapley value: the average number of rank places the target gains when that span is present. Every mention of the source's brand is tested together as one span.
The exact process is CHISEL (Chunk Holdout Influence Scoring by Elimination Loss), described in the paper CHISEL: Chunk Holdout Influence Scoring by Elimination Loss (PDF).
Single removals were tested first. Deleting one signal, the brand or one sentence never moved the top source's rank in our tests, while measuring each span across many combinations did.
In one example, the prompt "Are there trackers that compare desktop versus mobile Google search volatility?" was run on Google from Australia. The source Google cited first ranked 8th with its text removed and 3rd with its full text. One of its sentences was worth +4.10 ± 0.10 of those 5 places, and its brand name, mentioned four times, had no measured effect (+0.10 ± 0.10) (run 33).
DEJAN's Content Optimizer runs it in the Ablation tab. Full article: New Feature: Grounding Snippet Ablation.
