Inside Gemma 4 E2B: a 5-billion-parameter model that runs like 2.3B

Gemma 4 E2B-it is the smallest instruction-tuned model in Google DeepMind's Gemma 4 family, released in April 2026. It stores 5B parameters, computes with an effective 2.3B thanks to per-layer embeddings, reads text, images, video and audio in a 128K-token context, and fits in under 1.5 GB of memory. This post walks through what is inside it and how models like it are trained, with an interactive tool at every step. Everything here is aimed at a senior high schooler; the only math is multiplication, division and one exponential.

01The whole machine

Four kinds of input become vectors: a 262k-piece tokenizer chops text, and dedicated encoders convert pixels and sound. A stack of identical decoder layers then refines those vectors and emits one next token, which loops back as input. Click any block for a plain-English note, toggle the two attention modes, and see where the 2.3B-out-of-5B split comes from.

02How attention picks tokens

Every token gets three learned vectors: a query (what I am looking for), a key (what I offer), and a value (what I carry). The current token's query is dot-multiplied with every earlier key, the scores pass through softmax, and the output is a weighted average of values. Click any word to make it the query; later words are masked because during generation the future does not exist yet.

03Why per-layer embeddings make it cheap

An ordinary weight lives inside a matrix multiply, so every token touches it. A PLE parameter lives in a lookup table, and each layer fetches just the one row matching the current token. Compute therefore scales with the 2.3B that multiply, and the 2.7B of tables can wait in ordinary memory. Toggle the comparison against an imaginary model where all 5B multiply.

04What a lookup cannot do

A table row is fetched by token ID alone, so the row for "bank" is identical in "river bank" and "bank account". Rows store facts about a word in general and nothing about the sentence around it, which is why the dial cannot go to 100% tables. Google's own family shows the boundary: PLE appears only in the two phone-sized models, the big ones go dense or use Mixture-of-Experts.

05Mixture-of-Experts, the server-side sibling

The 26B model in the family stores 128 expert networks per MoE layer and activates 8 per token: 3.8B of 25.2B parameters. A tiny learned router scores all 128, keeps the top 8, softmaxes those scores into weights, and blends the expert outputs. Route the three tokens below and watch which cells wake up.

06How experts specialize without being told to

Experts only learn from tokens routed to them, and the router rewards experts whose output helped. Tiny random leanings therefore amplify: traffic follows skill, skill follows traffic. A balance penalty stops one early star from taking all the work. Drag the slider to replay the process on three experts.

07How the model was trained

Three acts: prepare data (a cleaned mix of web text, code, images and audio with a January 2025 cutoff), pretrain a predictor (guess every next token, corrected by gradient descent, with a larger Gemini-family teacher distilling its full probability distributions into the student), then post-train it into an assistant. Run training steps on one token yourself and watch the loss fall.

08What one weight feels

Gradient descent gives each weight one personalized nudge: new value = old value minus learning rate times slope, where backpropagation computes the slope of the loss with respect to that single number. Walk the ball downhill, then switch to the reckless step size and watch overshooting send the loss back up.

09Why the loss curve flattens

Cheap errors get fixed first, the leftovers are rare, and part of the loss can never be removed because language itself is uncertain. The flatness is also partly an axis illusion: on log-log axes the same curve is a straight line, meaning every tenfold of compute removes the same fraction of the remaining gap.

10Scaling laws, the planning equation

The straight line has a fitted formula. The 2022 Chinchilla version reads: loss = 1.69 + 406.4/N^0.34 + 410.7/D^0.28, a floor plus a too-small-model penalty plus a too-little-data penalty. Compute-optimal training lands near 20 tokens per parameter; phone models are overtrained far past that on purpose, because serving cost depends on model size alone. The sliders run the published equation.

11Where scaling strains

The measured line has held across six orders of magnitude, and nothing in the math breaks. The world objects instead: each tenfold costs ten times more, usable human text is estimated near 300 trillion tokens, and loss gains do not always translate into value. Ride the line and watch which wall trips first, then click the three escape routes the field is taking.

12The PLE bill, replayed per token

A second lens on the E2B trick: costs land per token in the matmuls and once per layer as a cheap fetch in the tables. Step a token through the stack, then set the matmul-versus-table split yourself. The real E2B sits at 54% tables, enough to punch above 2.3B, enough matmuls to still think.

13Attention by hand

The full computation with numbers small enough to check with a pencil: vectors of length 4, so the scale factor is exactly 2. Flip the query at the top and watch every downstream number change while keys and values stay put. In the real model the vectors are thousands of numbers long and 8 or more heads run at once, but the arithmetic is this.