Speculative Decoding Speedup Calculator
Inputs
| Acceptance rate | 70 % |
|---|---|
| Draft tokens per cycle | 5 |
| Draft cost ratio | 15 % |
Speculative Decoding Speedup Calculator
Estimate the decoding speedup from speculative sampling: enter the draft acceptance rate, draft tokens per cycle, and relative draft cost to get the expected accepted tokens and wall-clock speedup.
Inputs
Draft Model
Results
Enter a value to see results.
Details
Speculative Decoding Speedup
Speculative decoding speeds up language-model generation by letting a small, cheap draft model guess several tokens ahead, which the large target model then verifies all at once. When the guesses are right, the target produces several tokens for the price of a single forward pass; when they are wrong, it falls back to ordinary decoding. This calculator estimates the resulting speedup from three numbers: how often draft tokens are accepted, how many are proposed each cycle, and how expensive the draft model is relative to the target.
How it works
In a normal decoder the target model runs once per token. Speculative decoding instead asks the draft model to propose tokens, then runs the target once to check them in parallel. The target accepts each proposed token as long as it agrees with what the target itself would have sampled, stops at the first disagreement, and resamples that position from its own distribution. Because rejected tokens are corrected from the target, the output is statistically identical to plain decoding — the method trades extra draft compute for fewer expensive target passes without changing what the model says.
The formula
If each token is accepted independently with probability , the expected number of tokens produced per target pass is
τ=1−α1−αγ+1That would be the speedup if the draft model were free. Charging each draft pass a fraction of a target pass, the wall-clock speedup over ordinary decoding becomes
S=(γc+1)(1−α)1−αγ+1The numerator rewards acceptance and longer proposals; the denominator penalizes the draft work that those proposals cost.
Worked example
Suppose the draft model is accepted 70% of the time, proposes 5 tokens per cycle, and each draft pass costs 15% of a target pass:
τS=1−0.71−0.76=0.30.8824≈2.94=(5×0.15+1)(0.3)0.8824=0.5250.8824≈1.68Each target pass yields nearly three tokens on average, but after paying for five draft passes the realized speedup is about 1.7×. Raising acceptance to 90% would lift it well past 2×, while a heavier draft model — say 40% of the target's cost — would claw much of the gain back.
Picking the configuration
Two levers dominate. Acceptance rate rises when the draft model is closely aligned with the target, so drafts are usually smaller models from the same family or distillations of the target. Draft length has a sweet spot: because acceptance compounds, the probability that an entire run survives falls geometrically, so beyond a handful of tokens the extra draft cost outweighs the rare long acceptances. Sweeping the draft-token count here shows where the speedup peaks for a given acceptance rate. To see how the improved per-token speed feeds into end-to-end response time, see the Inference Latency Calculator.
Frequently Asked Questions (FAQ)
What is the acceptance rate?
After the draft model proposes a sequence of tokens, the target model checks them and accepts each one as long as it matches what the target would have sampled. The acceptance rate is the per-token probability of that match.
It rises when the draft model is well aligned with the target — for example a smaller version of the same family or a model fine-tuned to mimic it — and falls on hard or out-of-distribution text.
Crucially, speculative decoding is exact: rejected tokens are resampled from the target, so the output distribution is identical to ordinary decoding.
Why does the draft model’s cost reduce the speedup?
Every cycle runs the draft model several times to propose tokens and the target model once to verify them. If the draft model were free, the speedup would equal the expected tokens accepted per target pass. In reality each draft pass takes time, so the denominator adds the draft tokens multiplied by their relative cost. A draft model that is too large erodes its own benefit, which is why effective drafts are far smaller than the target.
How many draft tokens should be proposed per cycle?
There is a sweet spot. Proposing more tokens raises the best case, but because acceptance compounds, the chance that the whole run survives falls geometrically — once a token is rejected, every later proposal in that cycle is discarded. Past a handful of tokens the added draft cost outweighs the rare long acceptances. The optimum depends on the acceptance rate and the draft cost; trying a few values here shows where the speedup peaks.
Disclaimer
This is a simplified model that assumes a constant per-token acceptance rate and a fixed relative draft cost. Real acceptance varies with the text and the sampling temperature, and verification adds overhead not captured here, so measured speedups differ. Use the figure to compare configurations rather than as an exact prediction.