Last updated: 2026-06-22

Inference Latency

Inference latency is the time a user waits for a language model to answer. For a streaming completion it has two distinct parts: the initial pause before any text appears, and the steady rhythm of tokens that follows. This calculator combines both with the length of the answer to estimate the end-to-end response time and the throughput a single request actually experiences.

Time to first token and time per output token

Two measurements describe a streaming endpoint. Time to first token ($t_{ttft}$) is the delay from sending the request to seeing the first token. It covers reading the prompt, building the attention cache for every prompt token — the compute-bound prefill phase — and decoding one token. Long prompts lengthen it. Time per output token ($t_{tpot}$), also called inter-token latency, is the gap between each subsequent token during the decode phase. Decoding is memory-bandwidth bound: each new token reads the full set of model weights once, so this figure stays roughly constant and its reciprocal is the steady generation speed in tokens per second.

The formula

The first token is already included in time to first token, so each of the remaining tokens adds one inter-token gap:

where $N_{out}$ is the number of output tokens, $T$ is the total response time, and $v$ is the overall throughput. The minus-one term keeps the count exact; for long answers it barely changes the result.

Worked example

Suppose an endpoint reports a 200 ms time to first token and a 20 ms time per output token — a steady generation speed of 50 tokens per second — and the model produces a 500-token answer:

The answer takes about ten seconds, and the overall throughput of 49.1 tokens per second sits just below the raw 50 tokens per second because the fixed first-token delay is spread across the whole response. The gap between the two figures shrinks for longer answers and widens for short ones, where the startup pause dominates.

Reducing latency

Time to first token responds to shorter prompts, prompt caching, and faster prefill hardware. Time per output token responds to higher memory bandwidth, quantization, and techniques such as speculative decoding that emit more than one token per model pass. Because total time grows with every generated token, capping the maximum output length is often the most direct lever for interactive applications. To explore how draft-model acceptance changes the per-token figure, see the Speculative Decoding Speedup Calculator, and to model fleet-level capacity see the Effective Tokens per Second Calculator.