Last updated: 2026-06-22

Compute-Optimal Model Size

Given a fixed amount of training compute, the Chinchilla scaling laws answer how large a model that compute should build and how many tokens it should train on. The compute-optimal allocation pairs parameters and data in step rather than spending the whole budget on size. This calculator takes a compute budget in petaFLOP/s-days, holds the tokens-per-parameter ratio fixed, and inverts the 6ND rule to return the optimal parameter count and token count.

Allocating a compute budget

Training compute is well approximated by the 6ND rule: about six floating-point operations per parameter per token. The Chinchilla study showed that, for a fixed budget, loss is minimized when model size and dataset size grow together, with roughly twenty training tokens per parameter. Fixing that ratio turns the compute equation into a relation in the parameter count alone, which can be solved for the model size a given budget supports.

The formula

Let the budget be $C$ in petaFLOP/s-days and $r$ the tokens-per-parameter ratio. Converting the budget to FLOPs and substituting $D = rN$ into the $6\,N\,D$ rule gives $6 \cdot N \cdot rN = C_{\text{FLOPs}}$, so the parameter count and token count are

where $N$ is the optimal parameter count in billions and $D$ the optimal token count in billions. The factor $8.64 \times 10^{19}$ converts petaFLOP/s-days to FLOPs, and dividing by $10^{9}$ expresses $N$ in billions. The square root appears because compute grows with the square of the parameter count once tokens are tied to it.

Worked example

Take a budget of 6,000 petaFLOP/s-days at the central ratio of twenty tokens per parameter:

The budget is best spent on a model of about 65.7 billion parameters trained on roughly 1.31 trillion tokens. Doubling the budget would raise the optimal parameter count by about the square root of two, not by a factor of two, because of the square-root dependence.

Limits

This reflects compute-optimal training loss for a fixed training budget and nothing more. It does not weigh the cost of serving the model, which often makes a smaller model trained on more tokens the better practical choice, so deployed models frequently sit below the parameter count this returns. The fixed ratio is itself an estimate that shifts with dataset and architecture, and real runs sustain only a fraction of peak hardware throughput. The dataset side of the same trade-off is covered by the Chinchilla Optimal Tokens Calculator, and the raw compute of any chosen configuration by the Training FLOPs Calculator.