← back
New Feature: Grounding Snippet Ablation

New Feature: Grounding Snippet Ablation

The new Ablation tab in DEJAN's Content Optimizer measures how many rank places each sentence of an AI-cited source is worth.

Content Optimizer in the DEJAN app has a new Ablation tab, open to all users. It runs a grounded prompt with Google, OpenAI or Anthropic models, takes the sources the AI answer cited, and measures how many rank places each sentence or chunk of the top cited source is worth. In our public example, one sentence of the source Google cited first is worth 4.1 of the 5 rank places its text earns, and its brand name has no measured effect.

Example: desktop versus mobile volatility trackers

We ran the prompt "Are there trackers that compare desktop versus mobile Google search volatility?" on Google from Australia, in our public demo project. Every figure in this post is a screenshot of run 33, where you can check each number.

Three columns linking eight cited sources by grounding position, answer position and baseline rank, with semrush.com moving from 1st in grounding to 3rd in the baselineCited sources (Sources tab). Each line follows one source from the order Google grounded on, to the order it cited them in the answer, to the baseline rank our ranker gave it. The target, highlighted in blue, is grounded and cited 1st and ranked 3rd.

Google cited eight sources. The target is the one it cited first, Semrush's Sensor page. Our ranker placed it 3rd in all three baseline rankings, behind flyingvgroup.com and martech.zone.

With its text removed the target ranks 8th, and with its full text it ranks 3rd. One sentence is worth +4.10 ± 0.10 of those 5 places:

Semrush Sensor measures volatility in search results, tracking down 20+ categories on mobile and desktop and highlighting possible Google Updates.

The brand name, Semrush, appears four times and is tested as one span. It is worth +0.10 ± 0.10, which is inside its range of zero. The span that pulls the target down most is "You can use tools like Semrush's Traffic Analytics to guide your marketing strategies based on what works for your competitors." at −0.20 ± 0.13 places. The test took 188 rankings and the run cost $3.02.

Reading the results

The run page shows the result in seven views. Each figure below is one of them.

Horizontal bar chart of 20 spans and the brand, led by the Semrush Sensor sentence at plus 4.10 places, with all others between minus 0.20 and plus 0.20Rank places each span is worth (Main tab). Each bar is one span's Shapley value with its ± range. Green bars hold the rank up and red bars pull it down. A bar whose range reaches zero has no measured effect.

The target snippet's text with its first sentence in dark green, most other sentences in pale green, and four sentences in pale redHeat map (Text tab). The target's own text, coloured by value. The darker the green, the more rank places a sentence holds; red sentences pull the rank down. Brand mentions are underlined with dots and tested together.

Rows of bricks sized by word count, with the first brick in dark green and the rest in pale green or pale redValue map (Text tab). Each brick is one span, as wide as its word count, in text order. The dashed brick is the brand. Here the first brick holds most of the value.

Curve rising steeply to about 80 percent of the gains at 20 percent of the text, above a dashed diagonalHow concentrated the value is (Main tab). The share of the rank gain against the share of the text, with spans taken from most to least valuable. The dashed line is an even spread. In run 33, 20% of the text holds 80% of the gain.

Line chart of average rank rising from 8th with no text to about 3rd with full textRank by how much text is kept (Main tab). The target's average rank across rankings that kept the same number of spans, from no text on the left to the full text on the right. It rises from 8th to about 3rd.

Bar chart of how often each rank came up: 3rd 47 percent, 7th 37 percent, 8th 13 percent, 2nd 3 percent, 4th 1 percentHow often each rank came up (Main tab). The share of all 188 rankings that placed the target at each rank. Two peaks, 3rd at 47% and 7th at 37%, and almost nothing between them.

Grid of the latest 60 rankings, one row each, with filled squares for kept spans and the rank at the end of each row, alternating between 3rd and 7th or 8thRanking stream (Ranking stream tab). The latest 60 rankings, newest on top. Each row is one ranking and each column one span in text order, with the brand last. Empty squares are removed spans, and the label is the rank that ranking gave the target.

The two peaks come from the one sentence. In the 60 rankings in the stream, all 33 that kept it placed the target 2nd or 3rd, and all 27 without it placed the target 7th or 8th.

How it works

Step 1 runs the prompt, collects the text of each cited source, and ranks the unchanged set three times with the Content Optimizer ranker. The median is the baseline rank. Step 1 then cuts the target into spans. A span test typically costs between $0.50 and $20, depending on the size of the text, and never more than the cost cap. The default target is the source the platform cited first, and you can pick another.

Step 2 is the span test. It ranks the set many times with different combinations of the target's spans removed, and gives each span its Shapley value: the average number of rank places the target gains when that span is present. When the source names its brand, every mention is tested together as one span. The test stops when every ± range is 0.25 places or less, at the cost cap, or when every combination has been ranked.

We made the span test the default because in our tests, removing one thing at a time (a signal, the brand, a single sentence) did not move the rank. Measuring each span across many combinations did.

The method is called CHISEL, short for Chunk Holdout Influence Scoring by Elimination Loss. The paper, with results from 111 runs on Google, ChatGPT and Claude, is at dejan.ai/media/pdf/CHISEL.pdf.

Platforms

  • Google: the real snippet text the answer was grounded on (default), or a synthetic snippet.
  • OpenAI: the real snippet text the model read.
  • Claude: Claude does not return snippet text, so the app writes a synthetic extractive snippet from each fetched page.

Options

  • Mode: Snippet mode ranks the snippets. Page mode fetches each cited page and ranks the full pages.
  • Span unit: sentences or "..." parts in Snippet mode, paragraphs or sentences in Page mode.
  • Cost cap: the span test stops before it spends more than the cap.
  • Location: the country the prompt runs from, which changes the cited sources.
  • Signal test: the older method, in Advanced options. A model lists the target's signals and the app removes each one once.

Where to find it

Open a project in the DEJAN app and go to Content Optimizer → Ablation. Past runs are listed under the form. On pay-as-you-go projects both steps are billed from the credit balance, and step 2 starts only when the balance covers its cost cap.