Sign in
Listen

To understand how a single text feature changes an artificial intelligence model’s choice between two otherwise identical passages, we use a measurement method called Rank Signal.

In each test, we show the model two passages: one with the feature, and one without. We alternate their positions to prevent bias, and the model picks the passage it prefers. If it chooses the passage with the special feature, that counts as a win.

Instead of relying on a raw win rate, Rank Signal uses a Bayesian statistical model. Specifically, we apply a Beta-Binomial model with a Jeffreys prior. This approach gives us a posterior mean and a ninety-five percent credible interval.

Our decision rule is simple. If the entire interval sits above fifty percent, the feature is confirmed positive, meaning it helps. If the interval is completely below fifty percent, the feature is confirmed negative, meaning it hurts. If the interval still crosses fifty percent after forty trials, we conclude there is no measurable effect.

By using this Bayesian framework, we get a much more reliable picture than a simple percentage. For example, if a feature gets zero wins out of thirty-five trials, a raw win rate would show a flat zero percent. But our model factors in the prior, giving us a small, realistic non-zero figure instead. It is a highly rigorous way to see exactly what helps or hurts an AI model’s performance.

RANK SIGNAL · METHODOLOGY

The Bayesian method behind our rank-signal measurements

Rank Signal estimates how a single text feature changes an AI model's choice between two otherwise-identical passages. Each signal is a Beta-Binomial model with a Jeffreys prior, Beta(0.5, 0.5). The figure reported per signal is the posterior mean; the bar around it is the equal-tailed 95% credible interval, the 2.5th and 97.5th percentiles of the posterior, not a Wilson or Clopper-Pearson confidence interval. A signal's verdict is read from where that credible interval sits relative to 50%, the point at which the model has no preference between the two passages. The table below is an anonymized snapshot of 17 signals measured on Gemini 3.6 Flash across 3,206 paired decisions; the signal names are withheld.

Compute behind this sample: 3,973,582 input tokens and 984,371 output tokens across the runs that produced the data below.

One measurement

A single test shows the model two passages: one carrying the feature, one without it, identical in every other respect. The model picks the one it would use to answer, and a pick for the feature-carrying passage counts as a win. The side that carries the feature is alternated across tests so position cannot bias the result. Repeated over many independent passages, this gives w wins out of n trials.

The model

  • Prior: Beta(0.5, 0.5), the Jeffreys prior for a proportion.
  • Likelihood: Binomial, w wins in n independent trials.
  • Posterior: Beta(w + 0.5, n − w + 0.5).
  • Point estimate: the posterior mean, (w + 0.5) / (n + 1).
  • Interval: the equal-tailed 95% credible interval, from the 2.5th to the 97.5th percentile of that posterior.

The decision rule

  • Lower bound above 50%: confirmed positive, the feature helps.
  • Upper bound below 50%: confirmed negative, the feature hurts.
  • Interval still crossing 50% after 40 trials: no measurable effect.

No verdict is read before 20 trials. A second stage tests whether adding a positive feature flips a losing passage to a win against a competing passage; it requires at least 10 such tests and a flip rate of 30% or more. Every verdict in the table below is recomputed from the current data by this rule, so each row is consistent with its own interval.

Worked example

The strongest negative in the sample is S17: 6 wins in 185 trials. The posterior is Beta(6.5, 179.5), a mean of 3.5% and a 95% credible interval of [1.4%, 6.6%]. The whole interval lies about 43 points below the 50% line, so the verdict is confirmed negative.

The point estimate also shows the prior at work. A signal with 0 wins in 35 trials reports a posterior mean of (0 + 0.5) / (35 + 1) = 1.4%, not 0%. A raw win-rate would read a flat 0%; the small non-zero figure is the prior.

Sample data

17 signals measured on Gemini 3.6 Flash, pooled across all runs. Signal names are withheld. n is the number of paired decisions; the verdict is recomputed from the current interval by the rule above. The final column shows the second-stage competitor-flip test where it was run.

SignalVerdictnWinsPosterior mean95% credible intervalStage-2 flips
S01positive19019099.7%[98.7%, 100.0%]14 / 14
S02positive19519197.7%[95.2%, 99.3%]17 / 17
S03positive19618091.6%[87.4%, 95.1%]17 / 17
S04positive19117490.9%[86.4%, 94.5%]14 / 14
S05positive18716789.1%[84.3%, 93.1%]14 / 14
S06positive19816683.7%[78.2%, 88.5%]16 / 16
S07positive19414775.6%[69.4%, 81.4%]14 / 14
S08positive15411675.2%[68.1%, 81.6%]14 / 14
S09positive19813166.1%[59.4%, 72.5%]14 / 14
S10no effect1919851.3%[44.2%, 58.3%]14 / 14 *
S11no effect1959649.2%[42.3%, 56.2%]—
S12no effect1949347.9%[41.0%, 55.0%]—
S13negative1766637.6%[30.6%, 44.8%]—
S14negative1956734.4%[28.0%, 41.2%]—
S15negative1705431.9%[25.1%, 39.0%]—
S16negative1976231.6%[25.3%, 38.2%]—
S17negative18563.5%[1.4%, 6.6%]—

* S10 passed the second-stage flip test (14 / 14) during an earlier run, but its first-stage interval has since drifted to cross 50% as more trials accumulated. Its current verdict is therefore no effect. It is shown as a boundary case rather than removed.

Reproduce it

Given w wins and n trials, the posterior mean and 95% credible interval are:

from scipy.stats import beta
a, b = w + 0.5, n - w + 0.5
mean = a / (a + b)
lo, hi = beta.ppf(0.025, a, b), beta.ppf(0.975, a, b)

The anonymized dataset behind the table:

[
 {"signal":"S01","n":190,"wins":190,"verdict":"positive","mean":0.997,"ci":[0.987,1.000],"stage2":[14,14]},
 {"signal":"S02","n":195,"wins":191,"verdict":"positive","mean":0.977,"ci":[0.952,0.993],"stage2":[17,17]},
 {"signal":"S03","n":196,"wins":180,"verdict":"positive","mean":0.916,"ci":[0.874,0.951],"stage2":[17,17]},
 {"signal":"S04","n":191,"wins":174,"verdict":"positive","mean":0.909,"ci":[0.864,0.945],"stage2":[14,14]},
 {"signal":"S05","n":187,"wins":167,"verdict":"positive","mean":0.891,"ci":[0.843,0.931],"stage2":[14,14]},
 {"signal":"S06","n":198,"wins":166,"verdict":"positive","mean":0.837,"ci":[0.782,0.885],"stage2":[16,16]},
 {"signal":"S07","n":194,"wins":147,"verdict":"positive","mean":0.756,"ci":[0.694,0.814],"stage2":[14,14]},
 {"signal":"S08","n":154,"wins":116,"verdict":"positive","mean":0.752,"ci":[0.681,0.816],"stage2":[14,14]},
 {"signal":"S09","n":198,"wins":131,"verdict":"positive","mean":0.661,"ci":[0.594,0.725],"stage2":[14,14]},
 {"signal":"S10","n":191,"wins":98,"verdict":"no_effect","mean":0.513,"ci":[0.442,0.583],"stage2":[14,14]},
 {"signal":"S11","n":195,"wins":96,"verdict":"no_effect","mean":0.492,"ci":[0.423,0.562],"stage2":[0,0]},
 {"signal":"S12","n":194,"wins":93,"verdict":"no_effect","mean":0.479,"ci":[0.410,0.550],"stage2":[0,0]},
 {"signal":"S13","n":176,"wins":66,"verdict":"negative","mean":0.376,"ci":[0.306,0.448],"stage2":[0,0]},
 {"signal":"S14","n":195,"wins":67,"verdict":"negative","mean":0.344,"ci":[0.280,0.412],"stage2":[0,0]},
 {"signal":"S15","n":170,"wins":54,"verdict":"negative","mean":0.319,"ci":[0.251,0.390],"stage2":[0,0]},
 {"signal":"S16","n":197,"wins":62,"verdict":"negative","mean":0.316,"ci":[0.253,0.382],"stage2":[0,0]},
 {"signal":"S17","n":185,"wins":6,"verdict":"negative","mean":0.035,"ci":[0.014,0.066],"stage2":[0,0]}
]