FLOPs per Token Calculator
Inputs
| Pass type | Forward pass (2N) |
|---|---|
| Parameters | 70 |
FLOPs per Token Calculator
Estimate the floating-point operations a transformer performs per token using the 2N forward and 6N training rules, from the model’s parameter count in billions.
Inputs
Model
Results
Enter a value to see results.
Details
FLOPs per Token
How much computation does a language model spend on a single token? The answer underpins almost every back-of-the-envelope estimate in the field — training budgets, inference costs, and hardware utilization all start from it. The standard shortcut is remarkably simple: a dense transformer performs about two floating-point operations per parameter to process a token, and about six during training. This calculator applies those rules to a model's parameter count.
The 2N rule
A transformer's compute is dominated by large matrix multiplications, and in each of them every weight is touched once per token. A single multiply-accumulate counts as two floating-point operations — one multiplication and one addition — so pushing a token through a model with parameters costs roughly two operations per parameter, the 2N rule, on the forward pass. This is the estimate from the Kaplan scaling-laws work, and it has become the universal currency for reasoning about model compute. It deliberately ignores everything that is not a parameter-weighted matrix multiply, which is a good approximation for large models.
Forward and training cost
Training a model also requires a backward pass to compute gradients. The backward pass produces gradients with respect to both activations and weights, each about the cost of the forward pass, so it roughly doubles the forward work. The total is about three forward passes, or six operations per parameter per token (the 6N estimate):
GF=k⋅N(GFLOPs, N in billions)=k⋅N×109with the factor equal to two for a forward pass and six for a training step. Because is entered in billions, the forward figure lands directly in GFLOPs.
Worked example
For a 70-billion-parameter model on a forward pass:
GF=2×70=140 GFLOPs=2×70×109=1.4×1011 FLOPsEach generated token costs about 140 GFLOPs. Multiplying by a model's total throughput gives the achieved compute rate, and dividing that by an accelerator's peak gives its utilization. Training the same model would cost GFLOPs per token, and multiplying the training cost by the total tokens in a dataset is the usual way to size a training run's compute budget.
What it leaves out
The rule counts only the parameter-bearing matrix multiplications. The attention mechanism adds a separate term that grows with sequence length and does not depend on parameter count; for typical models at moderate context lengths it is small, but at very long contexts it can become significant. Embedding and normalization layers are likewise omitted. For these reasons the figure is an estimate for scaling and budgeting, not an exact instruction count. To see the achieved compute as a fraction of hardware peak, continue to the Model FLOPs Utilization Calculator.
Frequently Asked Questions (FAQ)
Where does the 2N estimate come from?
A transformer’s compute is dominated by matrix multiplications in which each weight is used once per token. A multiply-accumulate is two floating-point operations — one multiply and one add — so processing a single token through N parameters costs about 2N operations on the forward pass.
This is the rule of thumb introduced in the Kaplan scaling-laws work and reused throughout the field. It is an approximation: it counts the large matrix multiplies and treats everything else as negligible, which holds well for large models.
Why is training three times more expensive?
Training runs the forward pass and then a backward pass to compute gradients. The backward pass calculates gradients with respect to both the activations and the weights, each roughly the cost of the forward pass, so the backward pass is about twice the forward and the total is about three times — six operations per parameter per token.
Multiplying 6N by the total tokens in a dataset is the usual way to estimate the compute budget of a training run.
Does this include attention?
No. The 2N and 6N rules count only the parameter-bearing matrix multiplications. The attention mechanism adds a term that grows with sequence length and is independent of parameter count. For typical large models at moderate context lengths that term is small relative to the matrix multiplies, so the estimate stays close. At very long context lengths attention can become a significant share, and a more detailed model is needed.
Disclaimer
These are first-order estimates that count the dominant matrix multiplications and omit attention, normalization, and embedding terms. Actual operation counts depend on the architecture and context length. Use the figure for scaling comparisons and budget estimates, not exact hardware accounting.