LLM Eval Sample Size Calculator
Inputs
| Expected accuracy | 80 % |
|---|---|
| Margin of error | 5 % |
| Confidence level | 95% |
LLM Eval Sample Size Calculator
Find how many examples a model evaluation needs to measure accuracy within a target margin of error, using the standard sample-size formula for a proportion.
Inputs
Target Precision
Results
Enter a value to see results.
Details
LLM Eval Sample Size
When you measure a model's accuracy on a benchmark, the number you report is an estimate from a finite sample, and it carries uncertainty. Run the same model on a different fifty examples and the score will move. This calculator answers the planning question that comes before any evaluation: how many examples are needed so the measured accuracy lands within a stated margin of error at a chosen confidence level.
Accuracy is a proportion
A benchmark score is the fraction of examples a model answers correctly — a proportion. The uncertainty of an estimated proportion is described by its standard error, and a confidence interval is built by extending a number of standard errors on each side, set by the confidence level. Requiring that interval to be no wider than a target margin of error fixes the number of examples needed.
The formula
n=E2z2p(1−p)Here is the standard-normal value for the chosen confidence level, is the expected accuracy, is the margin of error, and is the number of examples, rounded up. Two features matter. The margin enters squared, so halving it quadruples the requirement — precision is expensive. And the accuracy enters through the product of and its complement, which peaks at an even split, so a model expected near a coin flip needs the most examples while one near the extremes needs fewer.
Worked example
Suppose you expect about 80% accuracy and want to know it to within ±5% with 95% confidence:
n=0.0521.962×0.8×0.2=0.00253.8416×0.16≈245.9→246So about 246 examples suffice. If you had no prior for the accuracy and used the worst case at an even split, the requirement would rise to 385. Tightening the margin to ±2.5% would push the worst-case figure to over 1,500 — a concrete illustration of the square-law cost of precision.
Using the result
The figure is a baseline that assumes independent examples and the normal approximation, both reasonable at evaluation scale. When you have no trustworthy estimate of accuracy, plan around the conservative 50% figure, since it is the largest the requirement can be. This calculator sizes a single accuracy estimate; deciding whether one model or prompt genuinely beats another is a comparison of two proportions that needs an effect size and statistical power, and usually more data — for that, see the A/B Test Significance Calculator. To reason about multi-sample code metrics instead, see the pass@k Calculator.
Frequently Asked Questions (FAQ)
Why does the sample size take this form?
Accuracy on a benchmark is a proportion: the share of examples the model gets right. The uncertainty in an estimated proportion follows the binomial standard error, which the formula inverts. To halve the margin of error you must quadruple the number of examples, because the margin shrinks with the square root of the sample size. Raising the confidence level enlarges the z-score and so the count.
The result assumes examples are drawn independently and uses the normal approximation, which is accurate for the sample sizes that evaluations typically need.
How does the expected accuracy change the answer?
The required size depends on p times one minus p, which is largest at 50% and falls toward the extremes. A model expected to score 95% needs fewer examples to pin down than one expected to score 50%, because near the extremes there is simply less variance to measure. If you have no reliable prior, use the conservative figure computed at 50% accuracy — it is the largest the requirement can be, so it is always safe.
Does this cover comparing two models?
No. This sizes a single accuracy estimate to a target precision. Deciding whether one model or prompt genuinely beats another is a comparison of two proportions, which needs both a desired effect size and statistical power, and generally requires more examples than estimating one model alone. For that, use a two-proportion test designed for win rates.
Disclaimer
The formula assumes independent examples and the normal approximation to the binomial. It sizes a single accuracy estimate, not a comparison between models, and ignores any clustering or repeated use of the same items. Treat the result as a planning baseline.