Pipeline-Parallel Bubble Calculator
Inputs
| Pipeline stages | 8 |
|---|---|
| Microbatches | 32 |
Pipeline-Parallel Bubble Calculator
Compute the pipeline bubble fraction and the resulting compute efficiency for pipeline-parallel training from the number of pipeline stages and the number of microbatches per batch.
Inputs
Pipeline
Results
Enter a value to see results.
Pipeline-Parallel Bubble
Pipeline parallelism splits a model into sequential stages and places each on a different device group, so a layer block on the first device hands its output to the next, and so on. It lets a model too large for one accelerator span several, but it pays a characteristic tax: the stages cannot all be busy at once. At the start of a batch the later stages sit idle waiting for data to reach them, and at the end the earlier stages fall idle as the last microbatches drain out. That idle interval is the pipeline bubble, and this calculator quantifies it.
Where the bubble comes from
Imagine an assembly line with eight stations. The first item must pass through stations one through seven before station eight has anything to do; by then seven steps have elapsed in which the line was not fully utilized. The same happens in reverse as the line empties. For a pipeline of stages this warmup-and-cooldown costs steps regardless of how much work follows. Whether that fixed cost matters depends on how many microbatches stream through behind it.
The formula
With pipeline stages and microbatches, the bubble fraction and the complementary efficiency for a standard synchronous schedule are
BE=m+p−1p−1=m+p−1mThe denominator is the total number of pipeline steps — the useful microbatches plus the steps of fill and drain. The two fractions always sum to one, since every step is either useful work or bubble.
Worked example
Take an 8-stage pipeline fed 32 microbatches per batch:
BE=32+8−18−1=397≈0.179=17.9%=3932≈0.821=82.1%Almost eighteen percent of the schedule is lost to the bubble, leaving about eighty-two percent for useful compute. Raising the microbatch count to 64 would drop the bubble to roughly eleven percent, since the fixed seven-step fill is now spread over more work.
Limits
The formula assumes the standard synchronous schedule, equal-cost stages, and communication between stages that is small enough to ignore. Interleaved schedules, which assign several non-contiguous stage chunks to each device, cut the bubble below what this gives, while uneven stage costs or heavy inter-stage transfers can make real efficiency worse. The microbatch count here is also the same quantity that, through the global batch, governs optimizer behaviour — see the Effective Batch Size Calculator — and pipeline parallelism is usually combined with data parallelism, whose distinct scaling loss is covered in the Data-Parallel Scaling Calculator.
Frequently Asked Questions (FAQ)
What is the pipeline bubble?
In pipeline parallelism the model is split into sequential stages on different devices, and microbatches flow through them like an assembly line. At the start of each batch only the first stage has work while the rest wait for data to arrive, and at the end the early stages fall idle while the last ones finish. That idle fill-and-drain time is the bubble.
For a synchronous schedule with the same number of forward and backward passes, the bubble is the stages-minus-one steps of warmup spread over the whole schedule.
How is the bubble reduced?
The direct lever is splitting the batch into more microbatches: the fill and drain cost is fixed at stages-minus-one steps, so dividing the batch more finely amortizes it over more useful work and shrinks the bubble fraction.
Interleaved and other advanced schedules cut it further by assigning several non-contiguous stages to each device. The trade-off is that more microbatches mean a smaller per-microbatch size, which can hurt kernel efficiency, so the bubble is reduced rather than eliminated.
How do stages and microbatches interact?
More stages deepen the pipeline, which raises the fill-and-drain cost and so enlarges the bubble for a fixed number of microbatches. More microbatches do the opposite, diluting that fixed cost. A useful rule of thumb is to keep the microbatch count well above the stage count — often by a factor of four or more — so the bubble stays small. The bubble fraction and the efficiency always sum to one, so improving either improves the other.
Disclaimer
This uses the standard synchronous pipeline bubble formula, which assumes equal-cost stages, a simple fill-and-drain schedule, and negligible communication between stages. Interleaved schedules and uneven stage costs change the result. Treat it as a planning estimate, not a guarantee of measured efficiency.