Epochs From Compute Budget Calculator
Inputs
| Compute budget | 6,000 |
|---|---|
| Parameters | 70 |
| Dataset tokens | 300 |
Epochs From Compute Budget Calculator
Find how many epochs a training compute budget can afford: convert PFLOP/s-days into trainable tokens with the 6N rule, then divide by your dataset size in tokens.
Inputs
Budget
Results
Enter a value to see results.
Epochs From a Compute Budget
A fixed compute budget and a fixed dataset together determine how many times a model can read its data. Planning a training run often comes down to that question: given so many PFLOP/s-days of allocation, a model of a known size, and a dataset of a known token count, how many epochs are affordable? This calculator answers it by converting the budget into trainable tokens with the standard cost model and dividing by the dataset size.
Compute, tokens, and epochs
Training compute is commonly allocated in PFLOP/s-days — one petaflop per second sustained for one day. The dominant cost of a training step is well approximated by six floating-point operations per parameter per token: two for the forward pass and roughly four for the backward pass. That rule turns a compute budget directly into a token count, because the budget in operations divided by the per-token cost is the number of tokens the budget can train on. Dividing those trainable tokens by the dataset size gives the number of full passes, or epochs.
The formula
Let be the budget in PFLOP/s-days, the parameters in billions, and the dataset in billions of tokens. One PFLOP/s-day equals operations, and billion parameters is at the six-operation rate, so the trainable tokens (billions) and the epochs are
Be=6⋅N⋅1018C⋅8.64×1019=DBWorked example
Take a budget of 6,000 PFLOP/s-days for a 70-billion-parameter model and a 300-billion-token dataset:
Be=6⋅70⋅10186000⋅8.64×1019≈1234.3 billion tokens=3001234.3≈4.11 epochsThe budget pays for about 1.23 trillion trainable tokens, which is roughly four full passes over the 300-billion-token dataset. A result this far above one is a signal that the dataset is small relative to the compute available.
Reading the result
An epoch count near one means the budget is matched to a single pass over the data, which is the regime compute-optimal recipes for large models tend to favor. A count well above one means there is enough compute to repeat the data several times; because fresh tokens are more informative than repeated ones, this often argues for gathering more unique data rather than cycling through the same corpus. A count below one means the budget cannot afford even a single pass, in which case a smaller model, a smaller data subset, or more compute is needed. The matched pairing of model size and token count for a budget is the subject of the Compute-Optimal Model Size Calculator.
Limits
The estimate uses the six-operations-per-parameter-per-token approximation and assumes the full budget is spent on useful training. Real runs sustain only a fraction of peak throughput, lose time to communication and pipeline overhead, and spend compute on evaluation and restarts, so the achievable epoch count is lower than this upper bound. The compute-optimal token target for a given model size, against which a dataset can be judged adequate, is given by the Chinchilla Optimal Tokens Calculator.
Frequently Asked Questions (FAQ)
How is the number of epochs derived from a compute budget?
A compute budget in PFLOP/s-days is first converted to raw floating-point operations: one PFLOP/s-day is 8.64 × 10^19 operations. The standard cost model charges about six operations per parameter per token for a full training step, so dividing the budget by six times the parameter count yields the tokens it can train on.
Dividing those trainable tokens by the size of the dataset gives the number of full passes — the epochs — the budget can afford.
Is training for more than one epoch advisable?
Compute-optimal recipes for large models favor a single pass over a very large, mostly unique dataset, because fresh tokens are more informative than repeated ones. When data is limited, repeating it for a few epochs can still improve a model, with returns that diminish as the same tokens are seen again.
If the calculator reports many affordable epochs, that usually signals the dataset is small for the budget and that gathering more unique data would use the compute more effectively than repeating what exists.
What does an epoch count below one mean?
A value below one means the budget cannot pay for a single full pass over the dataset at the chosen model size. The practical responses are to reduce the model size, since cost scales with parameter count, to use a smaller subset of the data, or to increase the compute budget. The compute-optimal model size for a given budget is a separate question addressed by the related model-size calculator.
Disclaimer
This uses the standard 6N FLOPs-per-token approximation and ignores overheads such as utilization below peak, attention terms, and evaluation. Real runs achieve a fraction of peak throughput, so treat the epoch count as an upper-bound planning figure.