Last updated: 2026-06-22

GPU Count for a Model

A large language model has to fit inside accelerator memory before it can serve a single token. When the model is bigger than one GPU, it is split across several, and the first planning question is simply how many devices it takes. This calculator answers that from three numbers: the total VRAM the model needs, the memory on each GPU, and the fraction of that memory a real deployment can actually use.

Why a model spans several GPUs

A modern accelerator carries a fixed amount of high-bandwidth memory — 80 GB on an H100, 141 GB on an H200. A model whose weights and runtime buffers exceed that figure cannot live on one device, so it is sharded. Tensor parallelism is the usual method: each layer is split across the group, every GPU holds a slice of the weights, and the partial results are stitched together with a collective communication step at each layer. The group then behaves as one larger device. The count this calculator produces describes that single tightly coupled group, not a set of independent replicas.

Usable memory is below the nameplate

A GPU never hands its full advertised memory to the model. The CUDA context, the framework's caching allocator, activation tensors, and fragmentation between allocations all take a share, and during serving the key-value cache grows with context length and batch size. Treating roughly 80 to 90 percent of the nameplate as usable leaves a realistic margin; pushing closer to the full figure invites out-of-memory failures under load.

The formula

With model requirement $M$ gigabytes, per-GPU memory $g$ gigabytes, and usable fraction $f$, the device count and total memory are

where $n$ is the number of GPUs and $T$ is their combined nameplate memory. The ceiling rounds up because a model cannot run on a fractional accelerator.

Worked example

Take a model needing 140 GB of VRAM on GPUs with 80 GB each, with 90 percent of memory usable:

Two GPUs suffice, with 160 GB of nameplate memory against a 140 GB requirement — comfortable headroom for the key-value cache. A 350 GB model on the same hardware needs $\lceil 350 / 72 \rceil = 5$ GPUs.

Beyond the minimum

The count here answers the fit question alone. Adding accelerators past that point buys memory headroom — a larger key-value cache, a higher batch size, room for longer contexts — and raises throughput rather than changing whether the model fits, while the communication overhead of sharding gradually erodes per-GPU efficiency. Sizing for performance is a separate exercise. To estimate the underlying memory requirement that feeds the $M$ input, see the LLM Inference VRAM Calculator, and to turn a device count into a running cost, the GPU Cloud Cost Calculator.