LoRA
Low-Rank Adaptation — a parameter-efficient fine-tuning technique that trains small adapter matrices instead of updating a model's full weights, dramatically reducing memory and compute requirements.
Fine-tuning a massive language model is incredibly resource-intensive. If you want to update a seven-billion-parameter model, the memory requirements can easily exceed eighty gigabytes of GPU space. That is where LoRA, or Low-Rank Adaptation, comes in.
Instead of modifying every single weight in a giant model, LoRA freezes the original weights entirely. It then inserts small, trainable adapter matrices into the model's attention layers. During training, only these lightweight adapters are updated. When it is time to run the model, the adapter outputs are simply added back to the original frozen layers. This approach delivers fine-tuning quality that is highly comparable to traditional methods, but at a fraction of the computational cost.
The key to tuning LoRA lies in its hyperparameters. The rank, or "r", determines the size of the adapter matrices. A lower rank means fewer parameters and faster training, while a higher rank offers more expressive power. Finding the right balance between rank and the scaling factor, known as alpha, is the secret to balancing efficiency and quality.
This approach makes advanced AI accessible on modest hardware. For example, developers used LoRA to transform a one-billion-parameter Gemma model into a compact, high-performing embedding model. Doing this with full fine-tuning on a single GPU would have been nearly impossible. With LoRA, it became entirely feasible, proving that you do not need massive computing power to build specialized, high-quality AI.
What LoRA is
LoRA (Low-Rank Adaptation) is a method for fine-tuning large models without updating all their weights. Instead of modifying every parameter in the model, LoRA inserts small trainable matrices — adapters — into the model's attention layers. The original weights are frozen; only the adapter matrices are trained. At inference time, the adapter outputs are added to the frozen layer outputs, producing a fine-tuned model without ever changing the base weights.
Why it matters
Full fine-tuning a large language model requires storing the gradient for every parameter — memory that scales with model size. A 7B-parameter model can require more than 80GB of GPU memory for a full fine-tuning run. LoRA reduces this to a fraction by training only the adapters, which may have a few million parameters rather than billions. The quality of the resulting model is often comparable to full fine-tuning, especially for task-specific adaptations.
DEJAN's use of LoRA
DEJAN used LoRA to fine-tune the Gemma 3 1B model into Gemma-Embed, a 256-dimension embedding model for semantic similarity and retrieval tasks. Full fine-tuning a 1B-parameter model for an embedding task would have been prohibitively expensive on a single GPU; LoRA made it feasible. The resulting model achieves strong embedding quality at a compact dimension that fits DEJAN's performance and storage requirements.
LoRA hyperparameters
The key LoRA settings are the rank (r) — the dimensionality of the adapter matrices, controlling how much capacity the adapters have — and the scaling factor (alpha). Lower rank means fewer parameters and faster training; higher rank gives the adapters more expressive power. Choosing rank is a hyperparameter decision that trades off between efficiency and adaptation quality.