Last updated: 2026-06-22

LoRA Parameters

Low-Rank Adaptation, or LoRA, is the most common way to fine-tune a large language model on a budget. Instead of updating every weight, it freezes the base model and trains a small pair of low-rank matrices alongside each adapted weight matrix. The result behaves almost like a full fine-tune while touching a tiny fraction of the parameters. This calculator estimates how many parameters that fraction amounts to, and how much optimizer memory it needs, from the model's shape and the adapter's rank.

How LoRA shrinks the trainable set

A transformer weight matrix maps one hidden vector to another, so for a square layer it holds about $d \times d$ values. Fine-tuning all of them is expensive: the optimizer must store gradients and running statistics for every one. LoRA replaces that dense update with a product of two thin matrices — one of shape $d \times r$ and one of shape $r \times d$, where the rank $r$ is far smaller than $d$. The base weights never move; only these two factors are trained, and at inference their product is added back into the original matrix.

Because each adapted matrix contributes $r \cdot (d_{\text{in}} + d_{\text{out}})$ trainable values — and $2 r d_{\text{model}}$ when input and output widths are equal — the trainable set is usually well under one percent of the full model. That is what makes LoRA practical on a single accelerator.

The formula

With $L$ layers, $m$ adapted modules per layer, rank $r$, and hidden size $d$, the trainable parameter count is

Mixed-precision Adam keeps roughly sixteen bytes of optimizer and master-weight state per trainable parameter — the 32-bit master copy, plus the first and second moment estimates — so the optimizer memory in megabytes is

This counts only the adapter state. The frozen base model still occupies memory for its weights, but it needs no gradients or optimizer state, which is where the saving comes from.

Worked example

Take a model with 32 layers, adapting the four attention projections in each layer, at rank 8 with a hidden size of 4096:

That is about 8.39 million trainable parameters. The optimizer state is

For a seven-billion-parameter base model, those 8.39 million adapters are roughly a tenth of a percent of the whole — yet they are enough to specialize the model for many downstream tasks.

Choosing the rank and modules

The rank is the main capacity dial. Ranks of 4 to 8 are common for light task adaptation; 16 to 64 give more room when the target task is far from the base model's training. Doubling the rank doubles both the trainable parameters and the optimizer memory, so it is worth raising only when a lower rank underfits. The modules-per-layer input reflects which matrices receive an adapter: two for query and value only, four for the full attention block, or more if the feed-forward matrices are included.

To compare the adapter against the size of the full model, see the Transformer Parameter Count Calculator. To size the total memory of a training run including the frozen weights and activations, see the LLM Training VRAM Calculator.