Last updated: 2026-06-22

LLM Inference VRAM

Serving a large language model starts with a blunt question: will it fit on the GPU? The answer is dominated by the model's weights, but the weights alone understate the requirement, because an inference server also has to hold the KV cache, the activations flowing through each layer, and some memory lost to fragmentation. This calculator estimates the total VRAM from three inputs — the parameter count, the precision the weights are stored in, and a runtime overhead that rolls the rest into one adjustable figure.

Weights set the floor

A model's weights are its largest fixed cost in memory. Each parameter is stored at a chosen precision: full 16-bit weights take two bytes each, 8-bit quantization takes one byte, and 4-bit takes half a byte. So a parameter count $N$ in billions, stored at $b$ bits, occupies

A billion 16-bit parameters is exactly 2 GB, which is why a 7B model needs 14 GB just for its weights. Dividing the bit count by eight converts bits to bytes, and because parameters are counted in billions and memory in gigabytes the units line up directly.

Adding runtime overhead

The weights are only the floor. During inference the server keeps a KV cache holding the attention keys and values for every token in flight, allocates activations as each request passes through the layers, and loses a little memory to fragmentation. The total is

where $o$ is the overhead expressed as a fraction of the weight size. The KV cache is the largest and most variable component — it grows with context length and the number of concurrent requests — so the right overhead depends heavily on the workload. Twenty percent is a reasonable starting point for short prompts at modest batch sizes; long-context, high-throughput serving can push it much higher.

Worked example

Take a 7-billion-parameter model served in 16-bit precision with 20 percent overhead:

The 16.8 GB total fits comfortably on a 24 GB card but leaves no room on a 16 GB one. To run the same model on a smaller GPU, dropping to 8-bit would halve the weights to 7 GB and bring the total near 8.4 GB. The relationship between model size and precision is explored further in the Model Size from Quantization Calculator.

Making a model fit

Because memory scales directly with the bits per weight, quantization is the most direct lever: 16-bit to 8-bit halves the weight memory, and 8-bit to 4-bit halves it again, usually with a modest and acceptable loss in output quality. When even a quantized model exceeds a single accelerator, the alternatives are sharding the weights across several GPUs or offloading part of the model to system memory at a steep speed cost. Estimating how many accelerators a model needs is covered in the GPU Count for Model Calculator. Treat the figure here as a planning estimate and confirm it against a real serving run, since the framework, context length, and batch size all move the true number.