Low-Rank Adaptation — a parameter-efficient fine-tuning technique that trains small adapter matrices instead of updating a model's full weights, dramatically reducing memory and compute requirements.
LoRA (Low-Rank Adaptation) is a method for fine-tuning large models without updating all their weights. Instead of modifying every parameter in the model, LoRA inserts small trainable matrices — adapters — into the model's attention layers. The original weights are frozen; only the adapter matrices are trained. At inference time, the adapter outputs are added to the frozen layer outputs, producing a fine-tuned model without ever changing the base weights.
Full fine-tuning a large language model requires storing the gradient for every parameter — memory that scales with model size. A 7B-parameter model can require more than 80GB of GPU memory for a full fine-tuning run. LoRA reduces this to a fraction by training only the adapters, which may have a few million parameters rather than billions. The quality of the resulting model is often comparable to full fine-tuning, especially for task-specific adaptations.
DEJAN used LoRA to fine-tune the Gemma 3 1B model into Gemma-Embed, a 256-dimension embedding model for semantic similarity and retrieval tasks. Full fine-tuning a 1B-parameter model for an embedding task would have been prohibitively expensive on a single GPU; LoRA made it feasible. The resulting model achieves strong embedding quality at a compact dimension that fits DEJAN's performance and storage requirements.
The key LoRA settings are the rank (r) — the dimensionality of the adapter matrices, controlling how much capacity the adapters have — and the scaling factor (alpha). Lower rank means fewer parameters and faster training; higher rank gives the adapters more expressive power. Choosing rank is a hyperparameter decision that trades off between efficiency and adaptation quality.
