Effective Batch Size Calculator
Inputs
| Per-device batch | 8 |
|---|---|
| Data-parallel devices | 64 |
| Accumulation steps | 4 |
| Sequence length | 4,096 |
Effective Batch Size Calculator
Compute the effective (global) batch size for distributed training from the per-device micro-batch, the number of data-parallel devices, and gradient-accumulation steps, plus tokens per optimizer step.
Inputs
Parallelism
Sequence
Results
Enter a value to see results.
Effective Batch Size
The effective batch size is the number of training examples that contribute to a single optimizer update. On one device it equals the micro-batch, but distributed training fans the work out across many accelerators and often sums several passes before each weight change, so the global batch the optimizer actually sees can be hundreds of times larger than what fits on a single chip. This calculator reconstructs that global figure, and the tokens it represents, from the three multipliers that produce it.
Why the global batch matters
Almost every learning-rate schedule, warmup length, and convergence result in the literature is stated in terms of the global batch, not the per-device micro-batch. A team that scales from 8 to 64 devices without recomputing the effective batch will silently change the optimization dynamics — the same code now takes a 8× larger step per update — and may diverge or stall. Knowing the effective batch is therefore a prerequisite for tuning the learning rate, reproducing a published recipe, or comparing two runs on different cluster sizes.
The formula
Three independent factors multiply together. With per-device micro-batch , the number of data-parallel replicas , and gradient-accumulation steps , the effective batch and the tokens per optimizer step at sequence length are
BT=b⋅d⋅a=B⋅LEach data-parallel replica processes its own micro-batch and the gradients are averaged, so replicas multiply the batch. Accumulation sums successive passes before one update, multiplying it again. The token count is simply the sequence count times the length of each sequence.
Worked example
Take a micro-batch of 8 sequences on each of 64 devices, with 4 accumulation steps and a 4,096-token context:
BT=8×64×4=2048=2048×4096=8,388,608The optimizer sees 2,048 sequences — roughly 8.4 million tokens — per update, even though no single device ever holds more than 8 sequences at a time. Doubling the accumulation steps to 8 would lift the effective batch to 4,096 without any change to per-device memory, at the cost of twice as many passes between updates.
Limits
This model covers data-parallel replication and gradient accumulation, the two factors that change the global batch. It deliberately ignores tensor, pipeline, and sequence parallelism: those split a single model or sequence across devices to fit memory, but they do not enlarge the global batch, so the devices among which a replica is sharded should not be counted in . The token figure also assumes a fixed sequence length; variable-length or packed batches need a separate token accounting. The memory side of choosing the micro-batch is covered in the Gradient Accumulation Memory Calculator, and how throughput scales as devices are added in the Data-Parallel Scaling Calculator.
Frequently Asked Questions (FAQ)
What is the effective batch size?
The effective, or global, batch size is the number of training examples that contribute to a single optimizer update. In distributed training it is the per-device micro-batch multiplied by the number of data-parallel replicas and by the gradient-accumulation steps, because every one of those passes is summed before the weights change once.
It is the figure that matters for learning-rate scaling and reproducibility, not the smaller micro-batch that fits on one device.
Why use gradient accumulation?
Gradient accumulation runs several forward and backward passes and sums their gradients before applying one optimizer step, which raises the effective batch without holding more activations in memory at once. It lets a memory-limited setup match the large-batch behaviour of a bigger cluster, at the cost of extra wall-clock time per update.
The trade-off is throughput for memory: each accumulation step is real compute, but only one optimizer update and one set of communication happens per cycle.
How do batch size and tokens per step differ?
Batch size counts sequences, while tokens per step counts the individual tokens those sequences contain — the batch multiplied by the sequence length. Token-budget planning and many published learning-rate schedules are stated per token, so the token figure is often the more portable one. Two runs with the same sequence count but different sequence lengths see different token counts per step, which is why both numbers are reported here.
Disclaimer
This calculator covers data-parallel replication and gradient accumulation only. It does not model tensor, pipeline, or sequence parallelism, which change how a global batch is split across devices but not the global batch itself. Confirm that your framework counts micro-batches and accumulation the same way.