Gradient Accumulation Memory Calculator
Inputs
| Micro-batch size | 4 |
|---|---|
| Accumulation steps | 8 |
| Data-parallel GPUs | 1 |
| Activation per sample | 0.5 |
Gradient Accumulation Memory Calculator
Compute the effective batch size reached through gradient accumulation and the activation memory it requires, from the micro-batch size, accumulation steps, data-parallel replicas, and per-sample activation cost.
Inputs
Batching
Memory
Results
Enter a value to see results.
Details
Gradient Accumulation Memory
Large language models train best with large batches, but a large batch means many samples held in memory at once — and activation memory is often the first thing to overflow a GPU. Gradient accumulation resolves the tension: it reaches a large effective batch while peak activation memory stays at the size of a single small micro-batch. This calculator shows both sides of that trade — the effective batch a configuration reaches, and the activation memory it actually holds.
What gradient accumulation does
Ordinarily a trainer updates the weights after every batch. With accumulation, it runs several micro-batches in sequence, adds up their gradients, and applies one optimizer step only once the target number of micro-batches has been processed. The summed gradient is identical to what a single batch of that combined size would have produced, so the update is mathematically the same — only its timing changes. The weights move once per accumulation cycle rather than once per micro-batch.
Why memory stays flat
Peak activation memory is governed by the largest forward and backward pass alive at one moment, and that is a single micro-batch. Accumulation processes one micro-batch at a time and keeps only a running sum of gradients, whose size matches the model's parameters and does not grow with the number of steps. So increasing the accumulation count raises the effective batch while leaving activation memory untouched. That is the entire point: a large effective batch on a memory budget that would never hold it all at once.
The formula
With micro-batch size , accumulation steps , data-parallel replicas , and per-sample activation cost in gigabytes,
EM=b⋅a⋅g=b⋅Awhere is the effective batch size and is the peak activation memory on one GPU. The effective batch multiplies across every replica and every accumulation step, while the memory term depends on the micro-batch alone.
Worked example
Take a micro-batch of 4 samples, accumulated over 8 steps on a single GPU, each sample costing 0.5 GB of activations:
EM=4×8×1=32=4×0.5=2 GBThe run trains as though the batch were 32, while only 2 GB of activations are ever resident. Spreading the same configuration across 4 data-parallel GPUs lifts the effective batch to , and the per-GPU activation memory is still 2 GB.
Reading the result
The effective batch size is the figure that learning-rate schedules and other hyperparameters are tuned against, not the micro-batch. Note that activation memory is only one part of the training budget: optimizer state, gradients, and the parameters themselves usually dominate, and this calculator deliberately isolates the activation term to make the batching trade-off visible. For the full memory picture see the LLM Training VRAM Calculator, and for shrinking the trainable parameter count with low-rank adapters, the LoRA Parameters Calculator.
Frequently Asked Questions (FAQ)
How does gradient accumulation work?
Instead of updating the weights after every micro-batch, the trainer runs several micro-batches in sequence, adds up their gradients, and applies a single optimizer step once the target number of micro-batches is reached.
The summed gradient is identical to what a single large batch of the same total size would produce, so the math of the update is unchanged — only the timing differs. The weights move once per accumulation cycle rather than once per micro-batch.
Why does memory stay flat as accumulation steps rise?
Peak activation memory is set by the largest forward and backward pass held in memory at once, which is a single micro-batch. Accumulation processes one micro-batch at a time and keeps only a running sum of gradients, whose size equals the model’s parameters and does not grow with the number of steps.
So doubling the accumulation steps doubles the effective batch while activation memory is unchanged — the central trade-off that makes large effective batches affordable on limited memory.
What does effective batch size mean?
The effective batch size is the number of samples that influence a single weight update. With data parallelism each replica contributes its micro-batch, and with accumulation each replica contributes several micro-batches in turn, so the effective batch is the product of the micro-batch, the accumulation steps, and the replica count. Learning-rate schedules and other hyperparameters are tuned against this effective figure, not the micro-batch.
Disclaimer
This estimate models activation memory from the micro-batch alone and omits optimizer state, gradients, parameters, and framework overhead, which dominate total training memory. Use it to reason about the batching trade-off, not as a complete memory budget.