Listen: Training Gemma‑3‑1B Embedding Model with LoRA

To automatically generate millions of search query suggestions, we need to translate vector embeddings back into natural language. But standard, off-the-shelf embedding models are designed for general similarity, not for being reversed back into text. To solve this, we built a custom embedding model called Gemma-Embed.

We created Gemma-Embed by fine-tuning a one-billion-parameter Google Gemma model. By using Low-Rank Adaptation, or LoRA, along with a custom projection head, we compress the embeddings into a consistent, two-hundred and fifty-six dimension space. Because we control this geometry, our query decoder can accurately map these vectors back into readable text.

Our training pipeline runs in three phases. First, we use unsupervised learning on more than half a million sentences to establish basic semantic relationships. Second, we train the model on millions of paraphrase pairs so it learns which sentences share the same meaning. Finally, we run in-domain self-contrast training using over seven million search queries.

This multi-stage process locks in a precise latent space. It allows us to successfully traverse the embedding space and generate diverse, high-quality search queries without relying on manual human labeling.