KV Cache Size Calculator
Inputs
| Layers | 32 |
|---|---|
| Key/value heads | 32 |
| Head dimension | 128 |
| Context length | 4,096 |
| Batch size | 1 |
| Precision | FP16 / BF16 (2 bytes) |
KV Cache Size Calculator
Estimate the key-value cache memory a transformer holds during inference, from layer count, key/value heads, head dimension, context length, batch size, and bytes per element.
Inputs
Model
Workload
Results
Enter a value to see results.
KV Cache Size
During transformer inference the model attends over every token seen so far. To avoid recomputing the key and value vectors of the whole context at each generation step, it stores them and reuses them — the key-value cache. That cache is often the dominant consumer of accelerator memory in long-context serving, sometimes larger than the model weights themselves. This calculator estimates its size from the layer count, the number of key/value heads, the head dimension, the context length, the batch size, and the bytes used per stored number.
Why the cache exists
Generating text one token at a time means each new token must attend to all the tokens before it. Without a cache, every step would reprocess the entire prompt and all earlier output, making generation cost grow with the square of the sequence length. By keeping the keys and values of past tokens, the model turns each step into work proportional only to the new token. The price of that speed is memory: one pair of vectors — a key and a value — for every layer, every key/value head, and every token in context.
The formula
The cache holds two tensors, keys and values, so the size in bytes is
Kbytes=2⋅L⋅H⋅d⋅s⋅B⋅ewhere is the number of layers, the number of key/value heads, the head dimension, the context length in tokens, the batch size, and the bytes per stored element. Dividing by gives gigabytes. The size grows linearly with each factor, which is the key intuition: doubling the context, the batch, or the layer count doubles the cache.
Grouped-query attention
The number of key/value heads is what grouped-query attention (GQA) targets. In plain multi-head attention every query head owns a key/value pair. GQA lets a group of query heads share a single key/value head, and multi-query attention shares just one across the whole layer. Since the cache scales with , dropping from 32 key/value heads to 8 cuts the cache to a quarter while the query heads — and most of the model's quality — stay in place. Most recent large models ship with GQA for exactly this reason.
Worked example
Take a model with 32 layers, 32 key/value heads, a head dimension of 128, holding 4,096 tokens of context, serving a single sequence in 16-bit precision:
Kbytes=2×32×32×128×4096×1×2=2,147,483,648which is about 2.15 GB. Serving four such sequences at once quadruples it to roughly 8.59 GB. Switching the cache to 8-bit precision would halve either figure.
Reading the result
Because every factor multiplies linearly, long-context, high-batch serving stacks them together and the cache can dwarf the weights. That is why production inference engines lean on cache quantization to fewer bytes per element, GQA to cut key/value heads, and paged attention to pack the cache without fragmentation. The compute side of the same hardware budget appears in the Attention Memory Calculator, and the model-weight portion in the LLM Inference VRAM Calculator.
Frequently Asked Questions (FAQ)
What is the KV cache?
During autoregressive generation a transformer attends over every previous token. Rather than recompute the key and value vectors for the whole context at each step, it stores them and reuses them — that store is the KV cache.
It trades memory for speed: without it, generating each new token would mean reprocessing the entire prompt and all earlier output. The cache holds two tensors, keys and values, for each layer and each key/value head, with one entry per token in context.
How does grouped-query attention reduce the cache?
Standard multi-head attention keeps a separate key/value pair for every query head. Grouped-query attention (GQA) lets several query heads share one key/value head, and multi-query attention takes this to the extreme of a single shared key/value head.
Because the cache size is proportional to the number of key/value heads, cutting them from, say, 32 to 8 shrinks the cache fourfold while leaving the query heads — and most of the model quality — untouched.
Why is long context so memory-expensive?
The cache grows linearly with context length: doubling the number of tokens in context doubles the cache. It also grows linearly with batch size, so serving many long-context sequences at once multiplies the two. At large context lengths the KV cache can exceed the memory taken by the model weights themselves, which is why long-context serving leans on techniques such as cache quantization, GQA, and paged attention.
Disclaimer
This estimate counts only the key and value tensors at the stated precision and ignores serving overhead such as memory fragmentation, paging metadata, and framework reservations. Actual usage on a given inference engine will be somewhat higher.