Inference Throughput Calculator
Inputs
| Parameters | 70 |
|---|---|
| Weight precision | 16-bit (2 bytes) |
| Memory bandwidth | 3,350 |
Inference Throughput Calculator
Estimate the memory-bandwidth limit on LLM decoding speed from model size, weight precision, and accelerator memory bandwidth — the tokens-per-second roofline for a single stream.
Inputs
Model
Hardware
Results
Enter a value to see results.
Details
Inference Throughput
Inference throughput is how many tokens a language model can generate per second. For a single request, the ceiling is set not by the accelerator's raw arithmetic power but by how quickly it can read the model's weights from memory. This calculator turns the model size, weight precision, and memory bandwidth of the hardware into that ceiling — the tokens-per-second roofline for one decoding stream.
Why decoding is memory bound
Generating text autoregressively means producing one token, then feeding it back to produce the next. Each of those steps has to read every weight in the model to compute a single new token. With a batch size of one, the accelerator performs very little arithmetic for each byte it loads, so it spends almost all of its time moving weights rather than multiplying. The token rate is therefore governed by memory bandwidth. (Prompt prefill is the opposite: it processes all prompt tokens together and is limited by compute.)
The formula
First find how much memory the weights occupy, then divide the available bandwidth by it:
Mv=N×b=MBWHere is the parameter count in billions, is the bytes stored per parameter, is the model size in gigabytes, is the memory bandwidth in gigabytes per second, and is the throughput limit in tokens per second. Because one billion bytes is about one gigabyte, entering parameters in billions makes the size land directly in gigabytes. The weight precision is 2 bytes for 16-bit weights, 1 byte for 8-bit, and half a byte for 4-bit.
Worked example
Take a 70-billion-parameter model in 16-bit precision served on an accelerator with 3,350 GB/s of memory bandwidth:
Mv=70×2=140 GB=1403350≈23.9 tokens/sSo a single stream tops out near 24 tokens per second. Quantizing the same model to 4-bit shrinks it to 35 GB and lifts the ceiling to about 96 tokens per second — a fourfold gain that mirrors the fourfold reduction in bytes read per token.
Reading the result
The figure is an upper bound, not a promise. It counts only the traffic of streaming the weights; the key-value cache, attention computation, and kernel overhead all eat into it, so a real single stream usually reaches somewhere between half and three-quarters of the roofline. Serving many requests in a batch reads the weights once for the whole group, pushing aggregate throughput far above the single-stream number — see the Batch Inference Throughput Calculator. To turn a token rate into a user-visible wait, pair it with the Inference Latency Calculator, and to gauge how well the hardware's compute is being used, see the Model FLOPs Utilization Calculator.
Frequently Asked Questions (FAQ)
Why is decoding limited by memory bandwidth rather than compute?
During autoregressive decoding the model generates one token at a time, and each step must read every weight from memory to compute the next token. With a batch size of one there is very little arithmetic per byte loaded, so the accelerator spends most of its time moving weights rather than multiplying.
The token rate is therefore set by how fast memory can be read, not by peak floating-point throughput. Prefill, which processes the whole prompt at once, is the opposite — it is compute bound.
Will a server actually reach this number?
No. The figure is a clean upper bound that assumes the only memory traffic is the weights. In practice the key-value cache, attention computation, kernel launch overhead, and imperfect memory utilization all subtract from it, so a single stream typically reaches somewhere between half and three-quarters of the roofline.
Batching many requests together raises aggregate throughput well above the single-stream limit because the weights are read once for the whole batch.
How much does quantization raise throughput?
Throughput scales inversely with model size, so halving the bytes per parameter roughly doubles the bound. Moving 16-bit weights to 8-bit halves the bytes read per token, and 4-bit halves them again. Quantization can reduce output quality, so the speed gain is weighed against any drop in accuracy for the task at hand.
Disclaimer
This is a first-order roofline that counts only weight memory traffic and assumes single-stream decoding. Actual throughput depends on the serving stack, batch size, sequence length, and key-value cache, and is usually a fraction of the bound. Benchmark on the target hardware before committing to a capacity plan.