A/B Test Significance Calculator
Inputs
| Wins (A) | 540 |
|---|---|
| Trials (A) | 1,000 |
| Wins (B) | 590 |
| Trials (B) | 1,000 |
A/B Test Significance Calculator
Test whether one prompt or model genuinely beats another. Enter wins and trials for each variant to get the win rates, a two-proportion z-test, the p-value, and the trials needed for power.
Inputs
Variant A
Variant B
Results
Enter a value to see results.
not_significant
Details
A/B Test Significance
When you change a prompt, swap a model, or tweak a decoding setting, the new version usually wins on some examples and loses on others. The question is whether the difference you measured reflects a genuine improvement or just the luck of which examples you happened to test. This calculator applies a two-proportion z-test to two win rates and reports the win rates, the test statistic, the p-value, and the number of trials a reliable comparison would need.
Comparing two win rates
Each variant is summarized by a win rate: wins divided by trials, where a "win" is whatever counts as success — a passed test, a preferred response in a head-to-head judgment, a correct answer. Two rates measured on finite samples will almost never be exactly equal even if the variants are identical, so the test asks how far apart they are relative to the noise in each estimate.
The formula
The test pools the two samples to estimate a common rate, uses it to compute the standard error of the difference, and divides:
pˉzP=na+nbwa+wb=pˉ(1−pˉ)(na1+nb1)pb−pa=2(1−Φ(∣z∣))Here and are the win rates, is the pooled rate, is how many standard errors separate the rates, is the standard normal cumulative distribution, and is the two-sided p-value. A magnitude of beyond roughly 1.96 marks the 5% significance threshold.
Worked example
Suppose prompt A wins 540 of 1,000 judged tasks and prompt B wins 590 of 1,000:
pˉzP=2000540+590=0.565=0.565×0.435×(0.001+0.001)0.59−0.54=0.02220.05≈2.26≈0.024The five-point gap clears the 1.96 threshold, with a p-value near 0.024 — below 0.05, so the difference is statistically significant: chance is an unlikely explanation. The planning figure shows that detecting a gap this size reliably (95% confidence, 80% power) would take about 1,538 trials per variant, so the 1,000 trials here put the result just over the line rather than comfortably past it.
Reading the result with care
Significance is not importance — a tiny, irrelevant gap can be significant with enough trials, and a meaningful gap can miss significance with too few. The test also assumes independent trials and a single comparison: evaluating many variants at once, or repeatedly checking results as data trickles in, both inflate false positives and call for adjustments. To size an evaluation before running it, see the LLM Eval Sample Size Calculator; for sample-and-check code metrics, see the pass@k Calculator.
Frequently Asked Questions (FAQ)
What does statistical significance mean here?
It asks whether the observed gap between the two win rates is larger than would routinely arise from chance if the variants were actually equal. The two-proportion z-test pools the two rates, estimates the standard error of their difference, and reports how many standard errors apart they are.
The p-value is the probability of a gap at least this large under the assumption of no real difference. A small p-value — conventionally below 0.05 — means chance is an unlikely explanation, so the difference is called significant.
It does not measure how large or important the difference is, only how confidently it can be distinguished from noise.
How many trials do I need?
Enough to detect the difference you care about without being fooled by noise. The calculator reports the trials per variant required to detect the observed gap at 95% confidence and 80% power — the standard planning targets.
Smaller true differences need many more trials, because the gap must clear a standard error that shrinks only with the square root of the sample size. If your current trial counts fall well below the figure shown, a non-significant result may simply reflect too little data rather than genuinely equal variants.
Is this a one-sided or two-sided test?
Two-sided. The p-value reflects the chance of a difference this large in either direction, which is the appropriate default when you did not commit in advance to which variant would win. A one-sided test, used only when the direction is fixed beforehand, would halve the p-value but risks missing a real effect in the unexpected direction. When comparing prompts or models without a prior commitment, the two-sided test is the safe choice.
Disclaimer
This test assumes independent trials and uses the normal approximation, which is reliable when each variant has at least a handful of wins and losses. It does not adjust for testing many variants at once or for repeated peeking at results, both of which inflate false positives. Treat it as evidence, not proof, and pre-register your comparison where possible.