Perplexity Calculator
Inputs
| Cross-entropy loss | 2 |
|---|
Perplexity Calculator
Convert a language model’s cross-entropy loss into perplexity and bits per token. Perplexity is the exponential of the average per-token loss measured on held-out text.
Inputs
Loss
Results
Enter a value to see results.
Details
Perplexity
Perplexity is the most widely used intrinsic measure of how well a language model predicts text. It summarizes the model's average uncertainty into a single number with an intuitive reading: roughly how many equally likely choices the model is weighing at each token. This calculator converts a cross-entropy loss — the quantity training frameworks report — into perplexity and into bits per token.
From loss to perplexity
When a language model reads held-out text, it assigns a probability to each actual next token. The cross-entropy loss is the average negative log of those probabilities; a confident, correct model earns a low loss, a surprised one a high loss. Perplexity is simply the exponential of that average:
PPLH2=eL=ln2Lwhere is the per-token loss in nats and is the equivalent in bits per token. Exponentiating undoes the logarithm inside the loss, turning an additive quantity into a multiplicative "effective vocabulary size." A perfect model assigns probability one to every correct token, giving a loss of zero and a perplexity of one — no uncertainty at all.
Worked example
Suppose a model reports a held-out cross-entropy loss of 2.0 nats:
PPLH2=e2.0≈7.39=ln22.0≈2.885 bits/tokenA perplexity of about 7.4 means that, averaged over the text, the model is as uncertain as if it were choosing uniformly among roughly seven or eight equally likely tokens. The bits-per-token figure connects to compression: an ideal coder using this model would need about 2.9 bits to encode each token, so a lower-perplexity model could compress the same text more tightly.
Reading and comparing perplexity
Lower perplexity means better prediction, but the number is only meaningful within a fixed tokenizer and evaluation set. Because it is measured per token, a model that breaks text into more, smaller tokens reports a different perplexity on the same passage than one with a coarser vocabulary, even at equal underlying quality. Comparisons are valid when the tokenizer and the test text are identical, and misleading otherwise. Perplexity also says nothing about whether a model can follow instructions or reason — for those, task-based evaluations are used. To plan how many examples such an evaluation needs, see the LLM Eval Sample Size Calculator, and for code-generation scoring see the pass@k Calculator.
Frequently Asked Questions (FAQ)
What does perplexity measure?
Perplexity measures how well a probability model predicts a sample — lower is better. It is the exponential of the average cross-entropy loss, and it can be read as an effective branching factor: a perplexity of 8 means the model is, on average, as uncertain as if it had to choose uniformly among 8 equally likely next tokens. A perfect model that always assigned probability one to the correct token would have a loss of zero and a perplexity of one.
Because it is computed directly from the model’s own probabilities, perplexity needs no labels and is the standard intrinsic measure of language-model fit.
What is the difference between nats and bits?
Both are units of information that differ only by the logarithm base. A nat uses the natural logarithm and is what most training code reports; a bit uses base two. One nat equals about 1.443 bits, so dividing the loss in nats by the natural log of two converts it to bits per token.
Bits per token is convenient because it connects directly to compression: a model with a lower bits-per-token figure could, in principle, compress the same text into fewer bits.
Can perplexity be compared across models?
Only when the tokenizer and the evaluation text are identical. Perplexity is measured per token, so a model that splits text into more, smaller tokens will report a different perplexity on the same passage than one using a coarser vocabulary, even at equal underlying quality.
Comparisons are therefore valid within a fixed tokenizer and dataset, and misleading across different ones. For capability comparisons across models, task-based metrics are used instead.
Disclaimer
Perplexity is an intrinsic measure tied to a specific tokenizer and evaluation set; it is not directly comparable across models with different vocabularies, nor a substitute for task-based evaluation. Confirm that the loss is per token and in nats before converting.