KV Cache Calculator for LLM Context and VRAM
Use a built-in open-weights model preset or the config.json values of any other model.
Key-value (KV) cache memory in bytes equals 2 × layers × KV heads × head dimension × tokens × sequences × bytes per value, and this KV cache calculator runs that formula for you. Running Qwen2.5 7B Instruct with 16-bit floating-point (FP16) KV precision at 32,768 tokens for 1 sequence requires 1.75 GB of KV cache memory. Added to 4-bit weights of 3.54 GB, the total required video random-access memory (VRAM) reaches 5.29 GB.
The KV cache grows with every token and every user, while the model weights stay fixed, so the calculator shows the two side by side. Enter your graphics processing unit (GPU) memory and the calculator also gives the longest context that fits.
KV Cache Calculator
✓ Fits on your 24 GB GPU. Longest context that fits in memory: 383,035 tokens per sequence at 1 sequence, more than the model's own 131,072-token limit, so the model is the ceiling.
KV cache = 2 × 28 layers × 4 KV heads × 128 head dim × 32,768 tokens × 1 sequence × 2 bytes, per NVIDIA's inference-optimization guide. Architecture from the Qwen2.5 7B Instruct model card. Activations and runtime buffers add more on top.
Reading Your KV Cache Calculator Result
The calculator displays output values as soon as you select a model preset or enter architecture fields. The calculator uses 1,024³ bytes per gigabyte and separates fixed model weights from the memory consumed by ongoing token generation.
KV cache, shown with a KB per token figure, is the memory that holds the attention keys and values for your context length. Model weights is the fixed memory the model takes at your chosen Weight quantization. Weights + KV cache adds the two, which is the GPU memory you need before activations and runtime buffers.
When you enter a value into Your GPU memory (GB, optional), the calculator shows whether the workload fits. The calculator also shows the Longest context that fits in memory, in tokens per sequence. If the longest context is longer than the model's own context limit, the calculator says the model's limit is the ceiling. You can click Copy result to copy the full specification breakdown as plain text.
How the Key-Value Cache Formula Works
Total key-value cache memory scales directly with transformer layer depth, key-value head counts, head dimensions, precision bytes, and active sequence tokens.
NVIDIA's developer blog tutorial on inference optimization documents that KV cache per token in bytes equals 2 × (num_layers) × (num_heads × dim_head) × precision_in_bytes. The factor of 2 accounts for separate key and value matrices. Total KV cache memory equals batch_size × sequence_length × that per-token figure.
Modern architectures use grouped-query attention (GQA) or multi-query attention (MQA) to reduce memory demand. In GQA, multiple query heads share a smaller number of key and value heads, so the calculator uses num_key_value_heads rather than total attention heads. NVIDIA's article works one example: a Llama 2 7B model in 16-bit precision, with batch size 1 and 4,096 tokens, has a KV cache of about 2 GB.
- num_layers: total transformer hidden layers in the model backbone.
- num_key_value_heads: distinct key and value projections per layer, which are lower than query heads in GQA models.
- head_dim: dimension size of each individual attention head.
- precision_in_bytes: memory cost per value, equal to 2 for FP16 or bfloat16 (BF16), 1 for 8-bit floating-point (FP8) or 8-bit integer (INT8), and 0.5 for 4-bit.
Context Length and Concurrent Users
Key-value cache memory scales linearly with context length and with concurrent users, so doubling either one doubles the KV cache. Static model weights remain constant regardless of sequence volume.
Qwen2.5 72B Instruct takes 320 KB per token in FP16 precision, so a single 32,768-token sequence requires 10.0 GB of KV cache memory. Paired with 33.85 GB of 4-bit weights, total memory reaches 43.85 GB. Serving 8 concurrent sequences at that same context length expands the KV cache to 80.0 GB, pushing total demand to 113.85 GB.
That concurrent 72B workload exceeds the capacity of an 80 GB GPU like the NVIDIA H100, and it exceeds the 94 GB memory of an H100 NVL. Set KV cache precision to FP8 / INT8 and the KV cache halves to 40.0 GB. The total drops to 73.85 GB, which fits in the 80 GB of one H100 before activations and runtime buffers.
Context length alone can push the KV cache past the size of the weights. Qwen2.5 32B Instruct at its full 131,072-token window needs 32.0 GB of FP16 KV cache for one sequence, more than twice its 15.13 GB of 4-bit weights.
Fitting Models to GPU Memory and Precision Levers
Type your card's memory into Your GPU memory (GB, optional) and the calculator says whether the model fits and how long a context it can hold. For example, Qwen2.5 14B Instruct requires 192 KB per token in FP16 precision. At 32,768 tokens, the KV cache consumes 6.0 GB, which adds to 6.85 GB of 4-bit weights for a total of 12.85 GB. This configuration fails on a 12 GB GPU but fits a 16 GB GPU, where the longest context that fits in memory reaches 49,997 tokens.
Lower the KV cache precision to free memory without changing the model. Moving from FP16 (2 bytes per value) to FP8 (1 byte per value) cuts key-value cache memory exactly in half. Quantizing the weights cuts a fixed amount of memory, while lowering the KV cache precision cuts the memory each token adds.
On Qwen2.5 32B Instruct at 131,072 tokens, switching to FP8 drops the KV cache from 32.0 GB down to 16.0 GB. Total baseline memory falls from 47.13 GB to 31.13 GB. On a 24 GB GPU, FP16 precision limits the longest context that fits in memory to 36,315 tokens, whereas FP8 precision extends that ceiling to 72,630 tokens.
Finding Architecture Values for Custom Models
To evaluate an architecture not included in the presets, select Custom (enter the config values) and expand the Model architecture (from the model's config.json) panel. The architecture values are in the model's config.json file on Hugging Face.
Read num_hidden_layers for Layers, num_key_value_heads for KV heads and head_dim for Head dim. Parameters (billions) is not in config.json, so take the total parameter count from the model card on the same Hugging Face page. If a configuration file omits head_dim, compute it by dividing hidden_size by num_attention_heads.
Each preset in the calculator uses the values from that model's config.json. Mistral Small 24B Instruct 2501 has 40 layers, 8 KV heads and head dim 128, or 160 KB per token. At 32,768 tokens that is 5.0 GB of KV cache, plus 11.18 GB of 4-bit weights, for 16.18 GB total. Microsoft Phi-4 has 40 layers, 10 KV heads and head dim 128, or 200 KB per token. Across its 16,384-token window that is 3.13 GB of KV cache, plus 6.52 GB of 4-bit weights, for 9.64 GB total.
Limitations of the KV Cache Calculator Estimate
The estimate excludes activations, runtime buffers, and framework memory overhead, meaning actual VRAM consumption during serving is higher.
Runtimes that use PagedAttention store the KV cache in fixed-size blocks to avoid memory fragmentation. Memory then grows a block at a time rather than token by token. FP8 and 4-bit KV cache only work if your runtime supports them. Without that runtime support, keep KV cache precision on FP16 / BF16 in the calculator.
If you are buying a machine and only need to know which GPU holds a model's weights, use the Local AI Hardware Calculator instead. For complete machines, see Best AI Workstations. For hosting choices, see How to Run Open-Weights Models.
With many users at once, the KV cache can outgrow the model weights. Qwen2.5 72B Instruct at 8 sequences of 32,768 tokens needs 80.0 GB of KV cache, more than twice its 33.85 GB of 4-bit weights. Run your model, context length and user count through the KV cache calculator above. Then leave room on the GPU for the activations and runtime buffers that the calculator does not count.
Frequently Asked Questions
- A KV cache is the GPU memory where a large language model (LLM) keeps the key and value tensors of every earlier token. In the decode phase the model generates one token per step, and each token depends on the keys and values of all previous tokens. Caching those keys and values avoids recomputing them at every step. The KV cache and the model weights are the two main contributors to an LLM's GPU memory requirement.
- For Qwen2.5 7B at FP16 precision (56 KB per token), a 131,072-token context consumes approximately 7.0 GB of KV cache memory. For Qwen2.5 32B (256 KB per token), that same context length requires 32.0 GB of KV cache memory in FP16 or 16.0 GB in FP8. The exact figure depends on the model's architecture and the KV cache precision.
- No. Weight quantization and KV cache precision are separate settings in this calculator. Quantizing model weights from 16-bit to 4-bit shrinks static storage on the GPU. The KV cache stays the same size unless you set a lower precision like FP8 or 4-bit for the cache itself.
- You can find this value in the model's config.json file in its repository under the num_key_value_heads property. In grouped-query attention models, this number is lower than num_attention_heads because multiple query heads share each key and value head.
- Each sequence holds its own keys and values for its own tokens, which is why the formula multiplies by the number of sequences. Doubling concurrent users or batch size therefore doubles total KV cache memory.
- PagedAttention stores the KV cache in fixed-size blocks to avoid memory fragmentation. The runtime allocates memory a block at a time, so usage moves in steps rather than token by token.
Need help sizing GPUs for an open-weights model?
A GPU that holds the model weights can still run out of memory once the context grows or more users connect. In an advisory engagement, Layer3Labs works through your model, context length and user count with you before you buy or rent GPUs.
Book a Consultation