Last updated: 2026-06-22

Training FLOPs

The compute required to train a large language model is dominated by a single quantity: the number of floating-point operations performed across the full training run. For dense transformers this is well approximated by the 6ND rule, which states that training costs about six operations for every parameter and every token seen. This calculator applies that rule, taking a parameter count and a token count and returning both the raw operation count and the equivalent in petaFLOP/s-days.

The 6ND rule

A transformer forward pass performs roughly two floating-point operations per parameter per token — one multiply and one add in the matrix multiplications that dominate the work. Training adds a backward pass that propagates gradients to both activations and weights, costing about twice the forward pass again. The forward and backward passes together come to roughly six operations per parameter per token. With $N$ parameters and $D$ training tokens, the total is therefore close to $6\,N\,D$. The estimate captures the dominant linear layers and omits smaller contributions from attention, embeddings, and normalization, which are negligible for large models.

The formula

With the pass factor $k$ (six for a training step, two for a forward pass only), parameter count $N$ in billions, and token count $D$ in billions, the total operation count and the compute in petaFLOP/s-days are

where $C$ is the total FLOPs and $P$ is the compute in petaFLOP/s-days. The factor of $10^{18}$ restores the billions in $N$ and $D$ to absolute counts. One petaFLOP/s-day is one petaFLOP/s sustained for a day, equal to $10^{15} \times 86{,}400 = 8.64 \times 10^{19}$ operations, so dividing by that figure converts a raw FLOP count into the unit large runs are usually reported in.

Worked example

Consider a 70-billion-parameter model trained on 1,400 billion tokens, a token-to-parameter ratio of twenty in line with compute-optimal practice:

The run needs about $5.88 \times 10^{23}$ operations, or roughly 6,806 petaFLOP/s-days. Switching the pass factor to two reports the forward-pass cost alone, one third of the training total.

Limits

The 6ND rule is an idealized count of arithmetic, not a prediction of wall-clock time or cost. Real hardware sustains only a fraction of its peak throughput, so the actual time follows from dividing this compute by the achieved rate rather than the advertised one — a gap captured by the Model FLOPs Utilization Calculator. The estimate also omits activation recomputation, optimizer steps, evaluation passes, and any runs that fail and restart, all of which add to the compute spent in practice. The companion question of how many tokens a model of a given size should see is addressed by the Chinchilla Optimal Tokens Calculator, and converting this compute into a rental budget by the GPU Cloud Cost Calculator.