Mapping Gemma embeddings to Gemini embeddings via a linear transform recovers entity vector structures with high fidelity without calling the paid API.
A single 768×768 matrix maps vectors from Google's free, local embeddinggemma-300m onto the output of the paid gemini-embedding-001 API. On 50,000 held-out Wikidata entities the matrix never trained on, the mapped vector sits at 0.83 cosine to the true Gemini vector, at no API cost.
Both models embed the same 199,998 Wikidata entities into 768 dimensions:
gemini-embedding-001, output truncated to 768 dims (a Matryoshka slice of its native 3072), un-normalized, with vectors averaging 0.586 in length.embeddinggemma-300m, run locally, unit-normalized by default.The two models share a family, a team, an overlapping tokenizer vocabulary, and a training lineage. Each entity is represented by one short label, not a document. Wikidata entities skew toward named entities.
Compared coordinate by coordinate, same-entity cosine between the two spaces is 0.009. The axes are rotated relative to each other: dimension 200 in Gemma does not correspond to dimension 200 in Gemini. Aligning the axes recovers the relationship.
A map fit from Gemma space to Gemini space on 120,000 entities, then measured on 50,000 disjoint held-out entities:
| Map type | Held-out cosine |
|---|---|
| none (raw) | 0.009 |
| orthogonal (rotation only) | 0.770 |
| linear (768×768) | 0.831 |
| affine (+ bias) | 0.831 |
A pure rotation recovers 0.770. Adding stretch and shear raises this to 0.831. Adding a bias term leaves it unchanged, so the relationship passes through the origin.
The map does not overfit: training-set cosine is 0.833, held-out cosine is 0.831. With 120,000 examples constraining 768 dimensions, the fit has 150 times more data than parameters. Ridge regularization from 0.001 to 1.0 all yields 0.832, and the score falls only above a strength of 10, so the setting changes the result by less than 0.001.
Both spaces are anisotropic: nearly every vector points toward one common direction. On a scale where 0 is uniform and 1 is all vectors identical, anisotropy is 0.48 for Gemini and 0.42 for Gemma. Part of the 0.770 rotation score comes from this shared tilt rather than from agreement about meaning.
Three alignment measures separate the two:
Whitening, which removes the dominant directions entirely, lowers alignment from 0.77 to 0.46. The shared structure therefore sits in the high-variance directions, not in smaller components that normalization would expose.
Centered similarity of 0.60 measures agreement about meaning once the tilt is removed. The 0.83 figure measures reconstruction of the full vector including the tilt, which is the quantity relevant to substituting one model's output for the other's.
Canonical correlation analysis measures the shared directions between the two spaces. The most-aligned direction correlates at 0.97. About 340 of 768 directions correlate above 0.5, and about 510 above 0.3. The remainder is model-specific.
The two models share a roughly 340-dimensional representation and each adds a few hundred model-specific dimensions. This matches the Platonic Representation Hypothesis, under which models trained on similar data converge toward a common geometry.
Clustering each space independently into 100 groups gives an adjusted mutual information of 0.49 between the two partitions, against a random baseline of 0.00.
Fitting the global rotation and reading the per-entity residuals shows which labels align and which do not.
Agreement is highest for proper names. Hungarian personal names reach 0.89 ("László Névy", "György Cséke"). Names carry little semantic content, so both models encode them from token surface form.
Agreement is lowest for rare technical strings: asteroid designations such as "(6627) 1981 FT", bare numbers, and obscure jargon. "Bacterial artificial chromosome" scores 0.49; "nephology" and "hamartiology" rank near the bottom. Labels containing digits average 0.739 against 0.774 for labels without. Non-Latin scripts average 0.787, above the 0.768 ASCII average.
Agreement is highest where encoding depends on surface form and lowest where it depends on semantic content.
Querying the Gemini index with mapped-Gemma vectors recovers 39% of each entity's true Gemini top-10 neighbors.
The predicted vector is at 0.833 cosine to the true vector. The true vector's 10th-nearest neighbor is at 0.694, and its nearest neighbor at 0.774. The prediction is closer to the target than any of the target's real neighbors, and falls inside the true top-10 shell for 99.7% of queries.
The true top-10 occupies a cosine band 0.08 wide, from 0.774 down to 0.694, shared with many other entities. A small angular error changes which ten rank highest. The true neighbors move to a median rank of 17, and recall reaches 78% at a search window of 100.
The map reconstructs vector geometry, and exact rank order does not transfer, because within a 0.08-wide band rank position carries little information. This holds for both models.
Gemini's 768-dimension embeddings are, to first order, a linear transform of Gemma's, reconstructable at 0.83 cosine from a 768×768 matrix fit on public data. Reconstruction supports clustering, similarity, and structural comparison. Exact ranked retrieval does not transfer.
Both embedding sets are public:
Dan Petrovic ·
Aug 14, 14:38