Model FLOPs Utilization Calculator
Inputs
| Workload | Inference (2N per token) |
|---|---|
| Parameters | 70 |
| Token throughput | 2,000 |
| Peak throughput | 989 |
Model FLOPs Utilization Calculator
Compute model FLOPs utilization (MFU): the share of an accelerator’s peak floating-point throughput a model run actually achieves, from parameter count, token throughput, and peak TFLOP/s.
Inputs
Workload
Hardware
Results
Enter a value to see results.
Details
Model FLOPs Utilization
Model FLOPs utilization, or MFU, measures how much of an accelerator's theoretical compute a model run actually uses. Two clusters with the same hardware can differ several-fold in real output, and MFU is the single number that captures that gap: it divides the floating-point operations a workload genuinely performs by the peak the hardware advertises. This calculator estimates it from the parameter count, the token throughput, and the accelerator's peak TFLOP/s.
What MFU tells you
Hardware vendors quote a peak rate — for a current data-center accelerator, on the order of a thousand 16-bit TFLOP/s — that assumes a perfect stream of fused multiply-adds with data already in registers. No real workload sustains that. MFU reports the fraction achieved, so an MFU of 40% means the run extracts forty cents of compute from every dollar of theoretical peak. It is the standard yardstick for judging whether a training or inference setup is well optimized, and for deciding whether a change actually helped.
The formula
A forward pass costs about two floating-point operations per parameter per token; a training step adds a backward pass for roughly six in total. With the workload factor (two for inference, six for training), parameter count in billions, and throughput in tokens per second, the achieved rate and utilization are
AU=1000k⋅N⋅v=PAwhere is the achieved TFLOP/s, is the peak TFLOP/s summed over every accelerator in the run, and is the utilization. The division by 1000 converts a billion parameters times tokens per second into TFLOP/s.
Worked example
Take a 70-billion-parameter model serving inference at 2,000 tokens per second on a single accelerator rated at 989 TFLOP/s of dense 16-bit compute:
AU=10002×70×2000=280 TFLOP/s=989280≈0.283=28.3%The run uses a little over a quarter of the chip's peak — typical for memory-bound decoding, where much of the time is spent reading weights rather than multiplying. A well-tuned large-model training job, which is more compute bound, would more often land between 35% and 55%.
Why it never reaches 100%
Real runs lose time to memory transfers, communication between accelerators, pipeline bubbles, and layers such as attention and normalization that are not pure matrix multiplies. A perfect utilization would mean none of that overhead exists. The practical use of MFU is comparative: measure it before and after an optimization on the same workload and hardware, and a higher number means more of the compute you are paying for is doing useful work. The per-token operation count that drives this estimate is explored in the FLOPs per Token Calculator, and the memory-bandwidth side of the same hardware in the Inference Throughput Calculator.
Frequently Asked Questions (FAQ)
What counts as a good MFU?
For large-model training, well-tuned systems typically report a model FLOPs utilization between roughly 35% and 55%; the PaLM work that popularized the metric reported about 46%.
Inference utilization is usually lower during memory-bound decoding because the accelerator spends much of its time reading weights rather than computing. There is no universal target — the useful comparison is against the same workload on the same hardware after an optimization, where a higher number means more of the paid-for compute is doing useful work.
Why is training six FLOPs per parameter but inference two?
A forward pass performs roughly two floating-point operations per parameter per token — one multiply and one add in the dominant matrix multiplications.
Training adds a backward pass that computes gradients with respect to both activations and weights, which costs about twice the forward pass, giving roughly six operations per parameter per token in total. These are the standard first-order estimates; they ignore attention and other terms that are small for large models.
Why can’t utilization reach 100%?
Peak throughput is a theoretical figure for back-to-back fused multiply-adds with data already in registers. Real runs lose time to memory transfers, communication between accelerators, pipeline bubbles, attention and normalization layers that are not pure matrix multiplies, and kernels that cannot perfectly fill the compute units. A utilization near one would mean none of that overhead exists, which never happens in practice.
Disclaimer
MFU uses first-order FLOP estimates (2N for inference, 6N for training) that omit attention and other minor terms, and assumes the peak figure matches the run’s precision and accelerator count. Treat it as a comparative efficiency indicator, not an exact accounting of every operation.