Last updated: 2026-06-22

pass@k

pass@k is the standard way to score language models on tasks where a candidate answer can be automatically checked — most famously code generation, where each attempt either passes a test suite or does not. Instead of asking whether a single answer is right, pass@k asks whether any of k attempts is right. This calculator turns a model's per-sample success rate into pass@k, so a single measured accuracy can be projected to any number of attempts.

What pass@k measures

For each problem in a benchmark, the model generates k independent candidate solutions, and the problem counts as solved if at least one candidate passes. pass@k is the fraction of problems solved under that rule. pass@1 is ordinary single-shot accuracy; pass@10 and pass@100 measure how often the model can reach a correct answer given several tries. The metric matters whenever an external verifier — a test suite, a compiler, a proof checker — can recognize a correct answer among many candidates, because then extra attempts translate directly into more solved problems.

The formula

If each sample succeeds independently with probability $p$, then a single sample fails with probability one minus $p$, and all $k$ samples fail with probability $(1-p)^k$. pass@k is the complement:

where $P_k$ is pass@k and $q_k$ is the probability that every attempt fails. The all-fail probability shrinks geometrically with $k$, which is why even a modest per-sample rate climbs toward certainty with enough attempts.

Worked example

Suppose a model solves a coding problem on 40% of single attempts, and the evaluation allows 10 samples:

A per-sample rate of 40% becomes a pass@10 of about 99%. The leap illustrates why pass@k figures rise so quickly with k, and why benchmark tables report several values of k side by side — the shape of that curve says more about a model than any single point.

How it is measured

Estimating pass@k by drawing exactly k samples is noisy. Benchmarks such as HumanEval instead draw a larger pool of $n$ samples, count the $c$ that succeed, and use the unbiased estimator $P_k = 1 - \binom{n-c}{k} / \binom{n}{k}$, averaging over every way to choose $k$ of the $n$ samples. The independence form used here matches that estimator when $p$ is taken as $c/n$, and is the right tool for projecting a known success rate to a chosen attempt budget. The form assumes equal, independent success across samples; at very low temperature samples become nearly identical and adding more helps little. To plan how many problems an evaluation needs for a stable estimate, see the LLM Eval Sample Size Calculator.