Finding Bard Inside Google's Gemma 4

Delete a few thousand specific neurons from one layer of Gemma 4 and it changes its answer to "What is your name?" from

My name is Gemma 4.

to this:

My name is Bard.

Bard was renamed Gemini in February 2024. The name survives inside a model released years later, dormant, and it sits directly beneath the current identity: partial removal of the evidence for "Gemma 4" surfaces it. The model stores several candidate identities at different strengths, and the strongest one wins at decode time. We found this while building and testing a drill-down pipeline that traces a behavior from a chat answer to individual weight vectors. This article covers the pipeline, the two methods that failed, the two that worked, and the open questions.

Setup

All experiments run locally against google/gemma-4-E2B-it (35 layers, d_model 1536, per-layer MLP widths of 6,144 in layers 0 to 14 and 12,288 in layers 15 to 34), held resident in memory as a mutable PyTorch model. A FastAPI hub supervises the model process and writes every generation to SQLite the moment it completes. A browser dashboard organizes the work as a pipeline: define a target behavior, localize it to a layer, then to a neuron set, then to weights.

The core operation is ablation: zeroing a component during the forward pass and comparing the output against an unmodified baseline. PyTorch hooks intercept a module's inputs or outputs at inference time, so the intervention lasts one generation and modifies no weights.

Definitions used below:

Residual stream. A transformer layer adds to its input rather than replacing it. Each layer reads the running 1,536-dimensional vector, computes an attention contribution and an MLP contribution, and adds both back. The answer is decoded from the accumulated sum. This additive structure makes ablation clean: one contribution can be deleted while everything else flows.

MLP neurons. Each layer's MLP expands the stream to a wider hidden space, applies a nonlinearity, and projects back down through a matrix called down_proj. Each hidden unit is a neuron. It has an activation (how strongly it fires on a given input) and an output direction (its column in down_proj, the vector it writes into the residual stream when it fires).

Marker. We score every ablation two ways. "Changed" means the output differs from baseline at all, which is noisy because trivial rewording counts. "Broken" means a chosen substring, here "Gemma", disappeared from the output. Downstream methods need the binary broken test; the changed test wasted hours on rephrasings before we separated the two.

Macro localization

The first pass ablates each layer's attention block, each layer's MLP, and each whole layer, one at a time: 105 configurations. The identity survives ablation of most of the network. It breaks when any one of these is removed: mlp L5, mlp L6, mlp L8, mlp L13, mlp L15, mlp L23, plus whole-layer skips at some of the same depths and the always-catastrophic layer 0. Attention ablations at those layers leave the name intact.

The cleanest single result, at layer 5:

The fact is written by MLPs in a chain of early and middle layers. Layer 5 became the drill site.

Two failed methods

Single-neuron sweep. We ablated each of layer 5's 6,144 MLP neurons individually and compared each generation to baseline. Zero outputs changed. Removal of any single neuron in the layer leaves the answer identical, token for token.

Activation ranking. We profiled which neurons fire hardest on this prompt (peak absolute activation over the answer span) and ablated the top k as a group. At k = 50 the name survived. At k = 1,000, one sixth of the layer and its loudest sixth, the output was byte-identical to baseline.

Both methods assume that causal importance shows up in a locally observable quantity: a neuron's individual effect, or its firing magnitude. For this fact, in this model, both assumptions fail. The identity is stored redundantly across thousands of quiet neurons, robust to the loss of any single member and to the loss of the thousand loudest. Firing magnitude measures volume. It carries no information about which direction a neuron pushes or whether the computation needs it.

Two methods that worked

Attribution patching. Score each neuron by activation multiplied by the gradient of the answer's log-probability with respect to that activation. This is a first-order estimate of how much the answer's probability drops if the neuron goes silent, and it includes indirect effects that flow through later layers. One forward pass and one backward pass score every neuron in every layer at once: roughly 276,000 neurons ranked in seconds. The global map immediately showed structure the activation profile missed. The top contributors to the identity answer sit in layers 8 through 34, away from layer 5's loud neurons.

Delta debugging (ddmin). Ranking cannot find a set whose members only matter jointly, so the distributed case needs a method with no locality assumption. ddmin is a 25-year-old algorithm from software testing: given a large set that causes an effect and a yes/no test, it bisects repeatedly to a minimal subset that still causes the effect. Our test is one generation: ablate the candidate subset, check whether "Gemma" vanished. The starting set is all 6,144 neurons of layer 5's MLP, which is known to break the identity. The cost is thousands of generations, so every step streams to SQLite and the job survives crashes and restarts. In an offline check, ddmin recovered an exact 3-neuron redundant cause planted among 6,144 in 129 tests. On the real model the cause is far less compact, which is itself a measurement.

The discovery

The step log of the bisection contains the finding. Each time the algorithm removes a large enough slab of layer 5's evidence, the model answers fluently, in first person, with the older name:

My name is Bard. I am a large language model, trained by Google.

Across the first ~3,300 bisection steps, 30 partial-ablation states produced a confident Bard identity, and they are concentrated exactly at the successful reductions: the states at the boundary where "Gemma" stops winning.

The reading consistent with all of this: the network stores multiple candidate identities at different strengths, and layers like L5's MLP supply evidence that makes "Gemma 4" outscore the rest at decode time. Remove enough evidence and the argmax falls to the next stratum down. The new identity was written over the old one, which remains in place underneath, like sediment.

Supporting observations:

The fallback skips a generation. We searched every saved output, thousands of ablation results, for the full ancestor lineage: Meena (Google's 2.6B conversational model from January 2020), LaMDA, PaLM, Bard, Gemini.

Bard appears 30 times. Gemini appears zero times, despite being the direct successor name and the far more prominent product. Our working hypothesis is that the strata reflect training-data frequency of first-person identity statements rather than corporate chronology. Years of web text said "I am Bard" in the first person. First-person "my name is Gemini" text is comparatively rare, and Meena, LaMDA, and PaLM never spoke as public personas at scale. The model's second-choice self is whichever identity the internet asserted most often in the first person.

The fact has a weight-level address. A neuron's output direction is a fixed column of down_proj, and the output vocabulary projection (the unembedding) assigns each token a fixed direction. Their dot product is a static, prompt-free measure: how hard this neuron pushes the "Gemma" logit per unit of activation. Ranking all ~276k columns this way names late-layer neurons that write almost straight at the "Gemma" token; the strongest, in layer 33, spends more than half its output norm on it. The same probe pointed at "Bard" is queued as follow-up.

Method summary

1. Define the target as a marker substring. Its disappearance is a clean binary test; "output changed at all" is noise.
2. Layer scan: ablate attention, MLP, and whole layer per layer. Cheap (3 × n_layers generations), and it identifies the component type and the layer set.
3. Attribution patching for the global neuron map: activation × gradient of the answer log-probability, one forward and one backward pass, all layers at once. Verify the ranking causally by group-ablating the top k at increasing k.
4. If top-k ablations fail to break the behavior, the cause is distributed. Switch to ddmin on a known-breaking superset. It is the only method in this list free of locality assumptions.
5. Ground the survivors in weights: project each implicated down_proj column onto the target token's unembedding direction. This converts "these neurons matter on this prompt" into "these weight vectors store the fact."
6. Persist every generation the moment it completes. Distributed-cause hunts run to thousands of generations, and our main finding was sitting in the step logs.

Further research

The bisection is still running as we publish, down from 6,144 neurons to under 2,000.

1. The minimal Gemma set. The size of the converged set is a direct measurement of the fact's redundancy.
2. The Bard direction in weights. If Bard-writing columns exist alongside Gemma-writing columns, shared neurons with opposite-sign usage would indicate the identity was overwritten in place; disjoint circuitry would indicate layering.
3. The missing Gemini stratum. Static probes for "Gemini", "LaMDA", "Meena", and "PaLM". Absence in the weights, beyond absence in outputs, would confirm the data-frequency hypothesis.
4. The cross-layer chain. MLPs 5, 6, 8, 13, 15, and 23 each carry the identity individually. Attribution maps conditioned on partial ablations should distinguish redundant copies from stages of a computation (retrieve, format, commit).
5. From ablation to editing. The weight probe names specific down_proj columns. A rank-one edit to those columns, in the ROME family of techniques, should rename the model permanently and surgically.
6. Identity as a probe of training data. If dormant identities are recoverable in Gemma, they are likely recoverable in any model whose lineage had public ancestors. "What does this model think its name is, at each stratum?" is a cheap, repeatable window into what instruction tuning overwrote.

All experiments were run locally against google/gemma-4-E2B-it with hook-based inference-time ablation; no weights were modified. Every generation referenced above is preserved in the run database.