LLM Training VRAM Calculator
Inputs
| Parameters | 7 |
|---|---|
| Memory per parameter | Mixed-precision Adam (16 bytes/param) |
| Activation memory | 0 |
LLM Training VRAM Calculator
Estimate the GPU VRAM needed to fully fine-tune a large language model from its parameter count, optimizer choice, and activation memory, using the mixed-precision Adam rule of 16 bytes per parameter.
Inputs
Model
Activations
Results
Enter a value to see results.
Details
LLM Training VRAM
Fully fine-tuning a large language model needs far more GPU memory than running it does, and the reason is the optimizer. Inference holds only the weights, but training must also keep a gradient for every parameter and the optimizer's running state, then add the activations stored during the forward pass. This calculator estimates the total VRAM for full fine-tuning from the parameter count, the optimizer's memory cost per parameter, and a separately entered activation figure.
The model states dominate
The largest fixed cost in training is the model states — the weights, their gradients, and the optimizer state. With parameters in billions and bytes per parameter, the model states occupy
M=N⋅B GBThe standard figure for mixed-precision training with Adam is 16 bytes per parameter, which breaks down as follows: two bytes for the 16-bit weight used in the forward pass, two for its 16-bit gradient, four for a 32-bit master copy of the weight kept for numerical stability, and four each for Adam's two moment estimates — the momentum and the variance. That is for every parameter. Cheaper optimizers shrink the optimizer-state terms: 8-bit Adam stores its moments in lower precision for roughly 12 bytes per parameter, and plain SGD with momentum needs about 8.
Activations are entered separately
The model states are fixed once the model and optimizer are chosen, but the activations are not. They are the intermediate values cached during the forward pass so the backward pass can compute gradients, and their size scales with the batch size and the sequence length rather than the parameter count. Because of that independence, this calculator takes activation memory as its own input and simply adds it:
V=M+Awhere is the activation memory in GB. Activation memory can be measured by running a single training step and reading peak usage, or cut sharply with activation checkpointing, which recomputes activations in the backward pass instead of holding them.
Worked example
Take a 7-billion-parameter model fine-tuned with mixed-precision Adam and, to start, no activation estimate:
MV=7×16=112 GB=112+0=112 GBThe model states alone are 112 GB — already beyond a single 80 GB accelerator. Adding a realistic 20 GB of activations brings the total to 132 GB, which makes the point plainly: even a 7B model, trivial to serve in 16-bit inference, cannot be fully fine-tuned on one mainstream GPU. The much smaller memory needed to merely run the model is covered in the LLM Inference VRAM Calculator.
Why LoRA changes the picture
Most of the 16 bytes per parameter is the gradient and optimizer state, and those exist only for the parameters being trained. LoRA freezes the base weights and trains small low-rank adapters, so gradients and optimizer state cover well under one percent of the parameters. The frozen weights still take memory, but stripping away the largest term is what lets a model that needs 132 GB for full fine-tuning instead fit on a single 24 GB card. The size of those adapters, and how rank controls them, is explored in the LoRA Parameters Calculator. Treat the figure here as a planning estimate and confirm it against a real training step, since temporary buffers and framework allocations move the true number.
Frequently Asked Questions (FAQ)
Where does the 16 bytes per parameter come from?
Mixed-precision training with the Adam optimizer keeps several copies of each parameter. Two bytes hold the 16-bit weight used in the forward pass and two more hold its 16-bit gradient. The Adam optimizer then keeps a 32-bit master copy of the weight (four bytes) plus two 32-bit moment estimates — the momentum and the variance — at four bytes each.
Adding those gives 2 + 2 + 4 + 4 + 4 = 16 bytes per parameter, which is the standard estimate for the model states in full fine-tuning.
Why does LoRA need so much less memory?
Full fine-tuning carries gradients and optimizer state for every parameter, which is the bulk of the 16 bytes per parameter. LoRA freezes the original weights and trains only small low-rank adapter matrices, so gradients and optimizer state exist for a tiny fraction of the parameters — often well under one percent.
The frozen base weights still occupy memory, but they need no gradient or optimizer state, which removes the largest term and lets large models fine-tune on a single GPU. The size of those adapters is explored in the LoRA parameter calculator.
How do I estimate the activation memory?
Activation memory is the storage for intermediate values kept during the forward pass so gradients can be computed in the backward pass. Unlike the model states, it scales with batch size and sequence length rather than the parameter count, so it is entered separately here.
It can be measured directly by running a single training step and reading peak memory, or reduced sharply with activation checkpointing, which recomputes activations in the backward pass instead of storing them. For a rough plan, start with zero to see the model-state floor, then add a measured figure.
Disclaimer
This is a first-order estimate of full fine-tuning memory. It omits temporary buffers, communication overhead, and framework-specific allocations, and treats activations as a single entered figure. Confirm against a real training step before sizing hardware.