Chinchilla Optimal Tokens Calculator
Inputs
| Parameters | 70 |
|---|---|
| Tokens per parameter | 20 |
Chinchilla Optimal Tokens Calculator
Find the compute-optimal number of training tokens for a model using the Chinchilla rule of about twenty tokens per parameter, with the resulting training compute in FLOPs.
Inputs
Model
Results
Enter a value to see results.
Details
Chinchilla Optimal Tokens
The Chinchilla scaling result reshaped how language models are sized. It showed that for a fixed compute budget the best accuracy comes not from making a model as large as possible, but from scaling parameters and training tokens together — pairing each parameter with roughly twenty tokens of training data. This calculator applies that ratio, taking a parameter count and returning the compute-optimal token count along with the training compute it implies.
The compute-optimal ratio
Before Chinchilla, the prevailing practice spent most of a compute budget on parameters and comparatively little on data, producing models that were undertrained relative to their size. The 2022 DeepMind study fit scaling laws across hundreds of runs and found that loss is minimized, for a given budget, when model size and dataset size grow in step. The empirical balance point places the optimal dataset at about twenty training tokens per parameter. A model with parameters should therefore train on about tokens.
The formula
With parameter count in billions and a ratio of tokens per parameter, the optimal token count and the associated training compute are
DC=N⋅r=6⋅N⋅D⋅1018where is the optimal number of training tokens in billions and is the training compute in floating-point operations. The compute follows the rule, which charges about six operations per parameter per token, and the factor of restores the billions in and to absolute counts.
Worked example
Consider a 70-billion-parameter model and the central ratio of twenty tokens per parameter:
DC=70×20=1400 billion tokens=1.4 trillion=6×70×1400×1018=5.88×1023 FLOPsThe compute-optimal dataset is about 1.4 trillion tokens, and training on it costs roughly operations. Raising the ratio — say to a hundred tokens per parameter — increases both the token count and the compute in direct proportion.
Limits
The twenty-to-one ratio minimizes training loss for a fixed training budget, but it deliberately ignores the cost of serving the finished model. A model that will answer many requests is frequently trained on far more tokens than Chinchilla-optimal, because a smaller model trained longer is cheaper to run for the same quality. The exact optimal ratio also shifts with the dataset, tokenizer, and architecture, so twenty is a central estimate rather than a constant. The complementary problem of sizing a model to a fixed compute budget is handled by the Compute-Optimal Model Size Calculator, and the raw compute of a chosen configuration by the Training FLOPs Calculator.
Frequently Asked Questions (FAQ)
What is the Chinchilla scaling result?
The Chinchilla result, published by DeepMind in 2022, found that earlier large models were undertrained: for a fixed compute budget, accuracy improves most when model size and training tokens are scaled together rather than spending the budget mostly on parameters.
The study fit scaling laws across hundreds of training runs and concluded that the compute-optimal allocation pairs each parameter with roughly twenty training tokens, a sharp departure from prior models trained at far lower ratios.
Why about twenty tokens per parameter?
The ratio falls out of the scaling laws fit to the training data. Under a fixed compute budget, loss is minimized when the marginal benefit of adding parameters equals the marginal benefit of adding tokens, and the empirical exponents in the Chinchilla study place that balance near twenty tokens per parameter. The exact figure depends on the dataset and architecture, so twenty is a round central estimate rather than a precise constant.
Is twenty tokens per parameter still the right target?
Twenty is the compute-optimal ratio for minimizing training loss at a fixed training budget, but it ignores inference cost. A model that will serve many requests is often trained on far more tokens than Chinchilla-optimal, because a smaller model trained longer is cheaper to run even if its training was not compute-optimal.
Several widely used models are trained at ratios of a hundred or more for this reason, so twenty is a baseline rather than a ceiling.
Disclaimer
The twenty-tokens-per-parameter ratio is a central estimate from the Chinchilla scaling laws and varies with dataset and architecture. It optimizes training compute alone and does not account for inference cost, which often justifies training smaller models on far more tokens. Use the result as a starting point, not a fixed rule.