Last updated: 2026-06-22

Dataset Token Count

Training and budgeting a language model start from a single quantity: how many tokens the dataset contains. Corpora, however, are usually described in words, documents, or gigabytes, not tokens, and the model only ever reads tokens. This calculator bridges that gap by converting a word count into an estimated token count with a tokens-per-word ratio, then scaling by the number of training epochs to give the total a model will process.

Tokens, not words

A token is the atomic unit a tokenizer produces when it splits text into subword pieces. Frequent words often become a single token, while rarer or longer words are divided into several, so a passage almost always contains more tokens than words. Because the mapping is learned by the tokenizer, the ratio between tokens and words is a property of the tokenizer and the text rather than a universal constant. Treating words and tokens as interchangeable understates the true count and, with it, the compute a run will need.

How the estimate works

The corpus size enters in millions of words, $W$. Multiplying by the tokens-per-word ratio $r$ gives the tokens in a single pass over the data. Repeating the corpus for $e$ epochs multiplies the work the model performs, so the effective training tokens are the one-pass total times the epoch count. A billions figure is derived from the millions total because training-set sizes are most often quoted in billions of tokens.

The formula

With $W$ in millions of words, ratio $r$, and $e$ epochs, the one-pass total $T$ (millions), its billions form, and the epoch-scaled training total $E$ (millions) are

The billions figure is simply $T\,/\,1000$.

Worked example

Take a 100-million-word corpus tokenized at 1.3 tokens per word and trained for three epochs:

One pass over the data is about 130 million tokens, or 0.13 billion. Training for three epochs means the model performs forward and backward passes over the equivalent of 390 million tokens, even though only 130 million are unique. The unique total describes the information available; the epoch-scaled total describes the work and therefore the compute.

Choosing the ratio

For English prose with a modern byte-pair tokenizer, roughly 1.3 tokens per word is a reasonable default, and about four characters per token is a related approximation. Code, mathematics, and non-Latin scripts tend to fragment into more tokens, raising the ratio. The dependable approach is to run the intended tokenizer on a representative sample and divide its token output by the word count of that sample, then use the measured ratio here. The inverse conversion, from a token budget back to readable length, is handled by the Tokens to Words Calculator.

Limits

The result is an estimate. The exact token count depends on the specific tokenizer, on formatting and whitespace, and on how documents are joined, none of which a single ratio captures perfectly. Counting epochs as additional tokens is correct for compute and scaling, but repeated tokens carry no new information and offer diminishing returns past a few passes. To check whether a dataset is large enough for a given model size, compare the one-pass total against the compute-optimal target in the Chinchilla Optimal Tokens Calculator.