Q.What is the KV cache in an LLM, in plain English?

TL;DR: Bytes per token = 2 x layers x KV heads x head size x 2 bytes (bf16). From the published configs: Qwen2.5-7B needs 56 KiB per token (1.75 GiB at 32,768 tokens), Mistral-7B 128 KiB, Qwen3-8B 144 KiB, Qwen2.5-72B 320 KiB (10 GiB at 32K). Cloudflare's GLM-5.3 deployment advertised 1,048,576 tokens of context on a KV pool that held 681,280, and 155 replies leaked other customers' text.

Every "1M-token context" on a spec sheet is really a promise about GPU memory. Once you can do the multiplication yourself, you can check that promise, size your own hardware, and stop paying for context you never use. We dug into this for Cloudflare's KV cache leak, and the math below is the tool you need.

Key numbers

What the model is actually caching

A transformer reads text as tokens. In each attention layer, every token is turned into three vectors: a query (what am I looking for), a key (what do I contain) and a value (what I pass on). To write token 1,001, the model compares its query against the keys of tokens 1 to 1,000 and blends their values.

Keys and values for old tokens never change. So instead of recomputing them every step, the engine stores them once. That store is the KV cache. It is the reason generation is fast after the first token, and the reason the first token of a long prompt (the "prefill") takes the longest: that is when the whole cache gets built.

The formula you can run on any model

KV bytes per token = 2 (K and V) x layers x KV heads x head size x bytes per number

Pull num_hidden_layers, num_key_value_heads and the head size (head_dim, or hidden_size / num_attention_heads) out of any model's config.json. bf16 or fp16 is 2 bytes; an 8-bit cache is 1. Multiply by your context length and by how many requests run at once.

Computed by Tensorust from each model's Hugging Face config.json, Oct 27 2026.

Worked example: Qwen2.5-72B serving 8 users at 32,768 tokens each needs 8 x 10 GiB = 80 GiB of cache, on top of roughly 145 GB of bf16 weights. That is why long context is priced the way it is. The weights are a fixed cost; the cache is the per-user meter.

Why most models now share KV heads

Older models gave every attention head its own keys and values. Grouped-query attention (GQA) lets several query heads share one KV head. Mistral-7B has 32 query heads and 8 KV heads, so its cache is a quarter the size it would be otherwise.

Same model, same context: GQA cuts the cache 4x. Tensorust calculation from the published config.

The industry's answer to "long context is too expensive" has not been to give up on long context. It has been GQA, 8-bit and 4-bit caches, paged memory (vLLM's PagedAttention) and prefix reuse. Every one of those is an engineering win that made more context cheaper for everyone.

Where it goes wrong

Serving engines like vLLM and SGLang carve GPU memory into pages and hand them out across requests. Pages are not zeroed between customers because zeroing costs speed. That is fine as long as the bookkeeping is right.

At Cloudflare it was not. The prefill pool held 681,280 tokens; the product promised 1,048,576. Prompts over the line got swapped for a placeholder instead of an error, the decode side read pages that still held someone else's session, and the customer got an HTTP 200. The SGLang bug was public as issue #30233 from July 6 2026; the fixed release (0.5.20) shipped September 18. The full timeline and our PyPI version data are here.

Common misconceptions

What you can do now

1. Open the config.json of the model you run and compute bytes per token with the formula above. Multiply by your max context and your batch size, and compare it to the KV pool size your engine logs at startup.
2. If you self-host on SGLang, run pip show sglang and upgrade to 0.5.22 or later, the version Cloudflare points to.
3. Set your server's max context to what the pool actually holds, and add a test that sends a prompt just over the limit and expects an error, not a 200.
4. Paying a provider for long context? Read why 429 errors happen next. Want to know what your LLM bill should be? The AI Cost Audit prices it against your real usage.

Related

• Cloudflare's KV cache leak: 155 replies, 62 days of a public bug
• Rust is the future of AI
• The real AI risk is a quiet cascade
• What is prompt caching?

FAQ

What is the KV cache in simple terms?

It is the model's scratchpad of everything it has already read. For each token, every attention layer stores a key and a value vector so the model does not recompute them when it generates the next token. It sits in GPU memory and grows with every token.

How much memory does the KV cache use?

Bytes per token = 2 x layers x KV heads x head size x bytes per number. Qwen2.5-7B in bf16 uses 56 KiB per token, so 1.75 GiB for a full 32,768-token context. Qwen2.5-72B uses 320 KiB per token, 10 GiB at 32K.

Why does a longer context cost more?

The cache is linear in tokens and it must stay in fast GPU memory for the whole request. Memory, not math, usually caps how many users a GPU can serve at once.

Can the KV cache leak data between users?

It can if the serving engine has a bug. Servers pool cache pages across requests and do not wipe them for speed. Cloudflare reported 155 replies that carried other customers' data from Aug 27 to Sep 9 2026, traced to a pool smaller than the advertised context plus an SGLang bug.

What is the difference between the KV cache and prompt caching?

The KV cache lives for one request. Prompt caching keeps the KV cache of a repeated prefix around between requests so the provider can skip recomputing it, which is why cached input tokens are billed cheaper.

Sources

All checked October 27, 2026.

• Cloudflare, KV cache exposure remediation, Oct 26 2026: 155 affected replies, dates, 1,048,576 vs 681,280 tokens, SGLang issue #30233, fix and recommended version.
• Hugging Face model configs: Qwen2.5-7B-Instruct, Qwen2.5-72B-Instruct, Mistral-7B-v0.1, Qwen3-8B.
• Ainslie et al., GQA: Training Generalized Multi-Query Transformer Models, 2023.
• Kwon et al., Efficient Memory Management for LLM Serving with PagedAttention, 2023.
• SGLang release history on PyPI.
• Images: cards and charts by Tensorust.