LLM VRAM calculator.

How the calculator works

GPU memory for running (not training) a model is three things added up:

• Weights = parameters × bits per weight ÷ 8. An 8B model in FP16 is 16 bits a weight, so about 16 GB in decimal units (14.96 GB in the unit below). Quantization stores fewer bits per weight.
• KV cache = 2 (keys and values) × layers × KV heads × head dim × context length × bytes per value × batch. It grows with every token of context, which is why a model that loads fine can still run out of memory on a long chat. Models with grouped-query attention (few KV heads), sliding windows or latent attention keep it much smaller; the calculator uses each model's real layout.
• Overhead: the CUDA context, compute buffers and activations. It depends on your runtime and settings, so it is a percentage you can change. 10% is our rule of thumb, not a measurement; check the real number with nvidia-smi once the model is loaded.

Units. GB on this page means 1024³ bytes (GiB), the unit nvidia-smi and llama.cpp report. That is also why a "24 GB" card holds 24 of them.

Example. Llama 3.1 8B at Q4_K_M with an 8K context: weights 4.6 GB + KV cache 1.0 GB, plus 10% overhead = 6.1 GB. The weights figure matches the 4.58 GiB file llama.cpp lists for that quant. Push the same model to its full 128K context and the KV cache alone becomes 16.0 GB (total 22.6 GB).

Sources

• Bits per weight for GGUF quants: llama.cpp quantize README (measured on Llama 3.1 8B; other models land within a few hundredths of a bit), checked Oct 23, 2026.
• Layers, KV heads, head dim, sliding windows, latent attention and context limits: each model's config.json on Hugging Face, checked Oct 23, 2026. Parameter counts: Hugging Face safetensors totals, except gpt-oss (21B, 117B) and DeepSeek-R1 (671B), which come from their model cards because their checkpoints are packed or include extra modules.
• Llama 3.2 1B: meta-llama/Llama-3.2-1B-Instruct config.json
• Llama 3.2 3B: meta-llama/Llama-3.2-3B-Instruct config.json
• Llama 3.1 8B: meta-llama/Llama-3.1-8B-Instruct config.json
• Llama 3.3 70B: meta-llama/Llama-3.3-70B-Instruct config.json
• Qwen3 4B: Qwen/Qwen3-4B config.json
• Qwen3 8B: Qwen/Qwen3-8B config.json
• Qwen3 14B: Qwen/Qwen3-14B config.json
• Qwen3 32B: Qwen/Qwen3-32B config.json
• Qwen3 30B-A3B (MoE): Qwen/Qwen3-30B-A3B config.json
• Qwen3.6 27B: Qwen/Qwen3.6-27B config.json
• Qwen3.6 35B-A3B (MoE): Qwen/Qwen3.6-35B-A3B config.json
• Qwen2.5 Coder 32B: Qwen/Qwen2.5-Coder-32B-Instruct config.json
• Gemma 3 4B: google/gemma-3-4b-it config.json
• Gemma 3 12B: google/gemma-3-12b-it config.json
• Gemma 3 27B: google/gemma-3-27b-it config.json
• Gemma 4 31B: google/gemma-4-31B-it config.json
• Gemma 4 26B-A4B (MoE): google/gemma-4-26B-A4B-it config.json
• Mistral 7B v0.3: mistralai/Mistral-7B-Instruct-v0.3 config.json
• Mistral Small 3.1 24B: mistralai/Mistral-Small-3.1-24B-Instruct-2503 config.json
• Phi-4 14B: microsoft/phi-4 config.json
• gpt-oss-20b (MoE): openai/gpt-oss-20b config.json
• gpt-oss-120b (MoE): openai/gpt-oss-120b config.json
• GLM-4.7-Flash (MoE): zai-org/GLM-4.7-Flash config.json
• DeepSeek-R1 671B (MoE): deepseek-ai/DeepSeek-R1 config.json
• GPU memory, checked Oct 23, 2026:
• NVIDIA GeForce compare page (RTX 30, 40, 50 series)
• NVIDIA L4
• NVIDIA L40S
• NVIDIA A100
• NVIDIA H100
• NVIDIA H200
• NVIDIA RTX PRO 6000

FAQ

How much VRAM do I need to run an LLM?

Add weights, KV cache and overhead. Rough guide at 4-bit (Q4_K_M) with an 8K context: an 8B model needs about 6.1 GB, a 32B model about 22.7 GB and a 70B model about 47.0 GB. Pick your model above for the exact number.

Can I run an 8B model on an 8 GB GPU?

Yes at 4-bit: Llama 3.1 8B at Q4_K_M with an 8K context needs about 6.1 GB. In FP16 it needs about 17.6 GB, so you want a 24 GB card for full precision.

What is the KV cache and why does context length matter?

While generating, the model stores a key and a value vector for every past token in every attention layer so it does not recompute them. That store grows linearly with context length and batch size. At long context it can be bigger than the weights.

Does a smaller KV cache precision help?

Yes. llama.cpp and vLLM can store the KV cache in 8-bit (about half of FP16) or 4-bit, which lets you fit longer context on the same card. Quality impact depends on the model, so test it on your own prompts.

Do mixture-of-experts models need less VRAM?

They run faster per token because only a few experts are active, but every expert still has to be loaded. Qwen3 30B-A3B uses about 3B parameters per token and still needs memory for all 30.5B.

Why does Gemma 3 use so little KV cache at 128K context?

Five of every six Gemma 3 layers only look at the last 1,024 tokens. At 128K context, Gemma 3 27B needs about 10.4 GB of FP16 KV cache instead of the 62.0 GB it would need if every layer cached the full context.

Can I split a model across two GPUs?

Yes. llama.cpp, vLLM and others split layers or tensors across cards, so two 24 GB cards can hold a model that needs about 45 GB. Each card adds its own overhead, so leave some margin.