Data-Parallel Scaling Calculator
Inputs
| Devices | 64 |
|---|---|
| Per-device throughput | 2,000 |
| Scaling efficiency | 90 % |
Data-Parallel Scaling Calculator
Estimate real training throughput across data-parallel devices from per-device throughput and scaling efficiency, with the ideal linear throughput and the achieved speedup over a single device.
Inputs
Cluster
Results
Enter a value to see results.
Details
Data-Parallel Scaling
Data parallelism is the most common way to train a model faster: replicate it across many accelerators, give each replica a slice of the batch, and average their gradients every step. In an ideal world throughput would rise in exact proportion to the device count. In practice it does not, because the gradient averaging that holds the replicas in sync costs time that grows with the cluster. This calculator separates the ideal linear throughput from the realistic figure after that loss, and reports the resulting speedup.
Ideal versus actual
The ideal throughput is what perfect scaling would give: every device contributes its full standalone rate with no overhead. The actual throughput is that ideal discounted by a scaling efficiency — a single fraction between zero and one that absorbs all the imperfections of running in parallel. An efficiency of 0.9 means the cluster delivers ninety percent of the theoretical maximum, so a 64-device job behaves like 57.6 perfectly scaling devices.
The formula
With device count , per-device throughput , and scaling efficiency , the ideal throughput , actual throughput , and speedup relative to one device are
IAS=d⋅v=I⋅E=d⋅EThe speedup is just the device count scaled by efficiency, because a single device runs at and the cluster runs at .
Worked example
Take 64 devices, each sustaining 2,000 samples per second on its own, with a measured scaling efficiency of 0.9:
IAS=64×2000=128,000=128,000×0.9=115,200=64×0.9=57.6The cluster processes 115,200 samples per second — a 57.6× speedup over a single device, not the 64× that perfect scaling would imply. The roughly 13,000 samples per second of difference is the cost of keeping the replicas synchronized.
Limits
The model condenses every source of scaling loss into one efficiency number that the user supplies; it does not predict that number from interconnect bandwidth, model size, or batch. Efficiency is also rarely constant — it typically falls as more devices are added, because the gradient all-reduce grows while per-device compute stays fixed — so a value measured at one cluster size should not be assumed at another. Pure data parallelism is assumed here; the batch implications of that replication are covered in the Effective Batch Size Calculator, and the distinct overhead of splitting a model across pipeline stages in the Pipeline-Parallel Bubble Calculator.
Frequently Asked Questions (FAQ)
What is scaling efficiency?
Scaling efficiency is the ratio of the throughput a cluster actually achieves to the throughput it would achieve if every added device contributed its full standalone rate. An efficiency of 0.9 means 64 devices behave like 57.6 ideal ones.
It folds together every source of imperfect scaling — gradient communication, synchronization waits, and uneven work — into a single fraction between zero and one, and is the standard way to report how well a training job parallelizes.
Why is scaling not linear?
Each optimizer step in data-parallel training requires an all-reduce that exchanges and averages gradients across every device, and that communication grows with the cluster while the useful compute per device stays fixed.
Add synchronization barriers, stragglers that hold up the group, and a shrinking compute-to-communication ratio as devices multiply, and the marginal device contributes less than the first. The result is sub-linear scaling, captured here by an efficiency below one.
How can scaling efficiency be improved?
The usual levers are raising the compute done between communications — larger per-device batches or gradient accumulation — so the fixed all-reduce cost is amortized over more work, and using faster interconnects or topology-aware collective algorithms to cut the communication itself.
Overlapping gradient communication with the backward pass, and reducing stragglers through balanced data sharding, also help. Efficiency that holds steady as devices are added is the sign of a well-tuned parallel setup.
Disclaimer
This is a first-order model that captures scaling loss as one efficiency fraction supplied by the user. It does not derive that fraction from interconnect bandwidth, model size, or batch, and assumes pure data parallelism. Measure efficiency on your own cluster rather than assuming a fixed value.