Effective Tokens per Second Calculator
Inputs
| Peak tokens per second per accelerator | 2,500 |
|---|---|
| Accelerators | 8 |
| Utilization | 70 % |
Effective Tokens per Second Calculator
Estimate the sustained token throughput of an inference fleet from per-accelerator peak speed, accelerator count, and utilization — the realistic capacity, not the single-stream peak.
Inputs
Fleet
Results
Enter a value to see results.
Details
Effective Tokens per Second
A single accelerator's peak token rate is a benchmark number captured at its best moment. The throughput a whole deployment actually sustains over a day is something else: traffic ebbs and flows, accelerators sit partly idle, and capacity is held back for spikes. This calculator estimates that sustained, fleet-level rate from the per-accelerator peak, the number of accelerators, and a utilization factor — the figure you should plan capacity against.
Peak versus sustained throughput
It is easy to overestimate capacity by summing peak numbers. The peak assumes every accelerator is saturated with full batches at every instant, which never holds across a real fleet over real time. Demand is uneven, load balancing is imperfect, and operators deliberately keep slack to absorb surges. The utilization factor folds all of that into one multiplier, turning an optimistic ceiling into a number you can promise to a workload.
The formula
veffTd=vg⋅G⋅u=veff⋅86400where is the per-accelerator peak in tokens per second, is the number of accelerators, is the utilization fraction, is the sustained throughput, and is the tokens served per day (there are 86,400 seconds in a day). Throughput scales nearly linearly with accelerator count when each one carries an independent share of traffic.
Worked example
Take a fleet of 8 accelerators, each benchmarked at 2,500 tokens per second under full batching, running at 70% utilization:
veffTd=2500×8×0.7=14,000 tokens/s=14,000×86400≈1.21×109 tokens/dayThe fleet sustains 14,000 tokens per second and can serve roughly 1.2 billion tokens a day. Note how far that sits below the naive peak of 20,000 tokens per second — the 30% of capacity lost to imperfect utilization is exactly the headroom that better batching and scheduling could reclaim.
Using it for planning
Compare the tokens-per-day figure against expected demand: divide the demand by this capacity to find how many fleets of this size are needed, then add margin for peaks above the average. Because output rises with both accelerator count and utilization, a target can be met either by buying more hardware or by raising utilization — the latter is usually the cheaper lever. The per-accelerator peak itself comes from a batched benchmark, explored in the Batch Inference Throughput Calculator, and the cost consequence of running below full utilization is covered in the GPU Utilization Cost Calculator.
Frequently Asked Questions (FAQ)
How is this different from peak throughput?
Peak throughput is what a single accelerator reaches at a moment when it is fully batched and saturated. Effective throughput is what a whole fleet sustains over time once the real world intrudes: traffic arrives unevenly, some accelerators sit partly idle, batches are not always full, and capacity is held in reserve for spikes.
Multiplying the per-accelerator peak by the accelerator count gives a theoretical ceiling; multiplying by utilization brings it down to a number you can actually promise. Capacity planning should use the effective figure, not the peak.
Where does the per-accelerator peak come from?
It is best measured by benchmarking the specific model on the specific hardware at the batch size you intend to run, because it depends on model size, precision, sequence length, and the serving stack.
As a sanity check, a memory-bandwidth roofline gives an upper bound for single-stream decoding, and batched serving multiplies that by the number of concurrent requests. Use a benchmarked figure when you have one and the roofline only as an optimistic ceiling.
How do I use this for capacity planning?
Compare the effective throughput, or the tokens-per-day figure, against your expected demand. If a workload needs a certain number of tokens per day, divide that by the per-day capacity to find how many fleets of this size are required, then add headroom for peaks above the average.
Because output scales with both accelerator count and utilization, you can meet a target either by adding hardware or by raising utilization through better batching and scheduling — the latter is usually cheaper.
Disclaimer
This assumes throughput scales linearly with accelerator count and that a single utilization factor captures all real-world losses. Networking, load balancing, uneven sequence lengths, and tail latency can lower sustained output further. Benchmark on the target hardware and keep headroom above the average demand.