Last updated: 2026-06-22

Batch Inference Throughput

Batched inference is how production language-model servers reach high utilization: instead of decoding one request at a time, they process many together. Because the model's weights are read once and reused for the whole group, the server's total output climbs sharply with batch size — but each individual request waits for the batch to finish. This calculator separates those two views, turning a measured batch completion time into aggregate throughput, per-request speed, and a request rate.

The batching trade-off

Autoregressive decoding is memory bound: each step reads every weight to produce one token. When a single request is in flight, almost all of that memory traffic serves just one user. Group several requests into a batch and the same weight read produces a token for every request at once, so the aggregate token rate rises with batch size while the bandwidth cost barely changes. The catch is latency — a request cannot leave until the batch step that contains it completes, so the per-request speed does not improve and may dip slightly as batches grow.

The formulas

From the batch size $B$, the output tokens per request $N_{out}$, and the measured batch completion time $t_b$:

where $v_{agg}$ is aggregate throughput, $v_{req}$ is per-request throughput, and $R$ is completed requests per second. The aggregate is simply the per-request speed multiplied by the batch size.

Worked example

A server decodes a batch of 32 requests, each producing 256 tokens, and the batch finishes in 5 seconds:

The operator sees more than 1,600 tokens per second flowing out of the server, but each user reads at about 51 tokens per second. Doubling the batch to 64 — if memory allows and completion time stays near 5 seconds — would roughly double the aggregate while leaving the per-request figure unchanged.

Choosing a batch size

Aggregate throughput keeps rising with batch size only while the work remains memory bound. Once the batch saturates compute or the key-value cache exhausts device memory, completion time grows faster than the batch and per-request latency degrades. The practical target is the largest batch that keeps per-request latency acceptable while memory still fits, found by measuring completion time at several batch sizes. Real servers also use continuous batching, where requests join and leave on every step rather than as a fixed group, which raises sustained throughput further. For the single-stream ceiling that sets $v_{req}$, see the Inference Throughput Calculator; for fleet-level capacity across many replicas, see the Effective Tokens per Second Calculator.