The KV cache is memory a model server spends on every token of every active request, and four config values set its size per token. It is why a machine that holds a model's weights can still run short of memory for a long prompt or a second user. We explain what the cache stores, estimate its size for one public model with a short script, and go through the vLLM settings that budget it and reuse it.
What the KV cache stores
The cache exists because a language model writes one token at a time and each new token needs all the earlier ones. The paper that introduced vLLM puts it this way: "the generation process of each new token depends on all the previous tokens in that sequence, specifically their key and value vectors." Those vectors come from the attention layers, and the paper notes that "each token has a set of key and value vectors across layers and attention heads within a layer."
Without a cache, the model would rebuild them for the whole sequence at every step. The Hugging Face Transformers documentation describes the waste and the fix: the model "performs the same computations each time", and "A KV cache stores these calculations so they can be reused without recomputing them." With the cache in place, each step computes keys and values for the newest token only and reads the rest from memory.
One property matters for everything that follows. The paper states that "the KV cache of one token depends on all its previous tokens", so the same word at a different position, or after a different prefix, has a different entry.
Sources: Kwon et al., Efficient Memory Management for Large Language Model Serving with PagedAttention, arXiv 2309.06180, sections 2.2 and 4.1; Hugging Face Transformers documentation, Cache strategies.
What its size scales with
Since every token keeps vectors in every layer, the cache grows along several axes at once. The paper gives the arithmetic for its example, a 13B-parameter OPT model: "the KV cache of a single token demands 800 KB of space, calculated as 2 (key and value vectors) × 5120 (hidden state size) × 40 (number of layers) × 2 (bytes per FP16)." It then scales that up twice:
- By sequence length: at 2,048 tokens, "the memory required to store the KV cache of one request can be as much as 1.6 GB."
- By concurrent requests: "The KV Cache size grows quickly with the number of requests."
Newer models change one term.
Grouped-query attention lets several query heads share one key and value head, and the paper that proposed it points out that fewer key and value heads mean a smaller cache: going to a single shared head reduces "the size of the key-value cache and therefore amount of data that needs to be loaded by a factor of H", the number of heads.
A model's configuration file reports the count as num_key_value_heads, which the Transformers documentation describes as "the number of key_value heads that should be used to implement Grouped Query Attention."
vLLM's source shows the same terms from the implementation side.
For one attention layer, a cache page holds by default, for each KV head and each token in the block, (self.head_size + self.head_size_v) * get_dtype_size(self.dtype) bytes: a key and a value of one head's width.
Its Llama model code reads the KV head count from num_key_value_heads, falling back to num_attention_heads, and reads the head width from head_dim when the configuration has one, otherwise dividing the hidden size by the attention head count.
That gives five factors: layers, KV heads, head width, bytes per value, and tokens held.
Sources: Kwon et al., PagedAttention, section 3; Ainslie et al., GQA: Training Generalized Multi-Query Transformer Models from Multi-Head Checkpoints, arXiv 2305.13245, section 2.2; Hugging Face Transformers documentation, Qwen3MoE; vLLM v0.33.0 source, kv_cache_interface.py and llama.py, the head width read and the KV head count read.
Estimating the size for one model
Those five factors are enough for an estimate from public numbers, before any hardware is involved.
We use Qwen/Qwen3-Coder-30B-A3B-Instruct, whose config.json declares 48 layers, 4 KV heads, a head_dim of 128 and the bfloat16 data type.
We downloaded the file at a pinned commit, so the numbers cannot move under us:
curl -sSo config.json https://huggingface.co/Qwen/Qwen3-Coder-30B-A3B-Instruct/raw/b2cff646eb4bb1d68355c01b18ae02e7cf42d120/config.json
The script multiplies the factors and prints the result at a few context lengths:
"""Our estimate of KV cache size from a model's config.json (not vLLM's allocator)."""
import json
import sys
BYTES_PER_VALUE = {"bfloat16": 2, "float16": 2, "float32": 4}
with open(sys.argv[1], encoding="utf-8") as f:
cfg = json.load(f)
layers = cfg["num_hidden_layers"]
kv_heads = cfg.get("num_key_value_heads", cfg["num_attention_heads"])
head_dim = cfg.get("head_dim") or cfg["hidden_size"] // cfg["num_attention_heads"]
width = BYTES_PER_VALUE[cfg["torch_dtype"]]
per_token = 2 * layers * kv_heads * head_dim * width # 2 = one key and one value
print(f"layers={layers} kv_heads={kv_heads} head_dim={head_dim} dtype={cfg['torch_dtype']} ({width} bytes)")
print(f"bytes per token: {per_token:,} ({per_token // 1024} KiB)")
for tokens in (8_192, 32_768, 131_072, cfg["max_position_embeddings"]):
size = per_token * tokens
print(f"{tokens:>7,} tokens: {size:>14,} bytes = {size / 2**30:5.2f} GiB")
We ran it in a throwaway container with no network and a read-only filesystem:
$ docker run --rm --network none --read-only -v "$PWD":/demo:ro -w /demo python:3.12-slim python kv_cache_size.py config.json
layers=48 kv_heads=4 head_dim=128 dtype=bfloat16 (2 bytes)
bytes per token: 98,304 (96 KiB)
8,192 tokens: 805,306,368 bytes = 0.75 GiB
32,768 tokens: 3,221,225,472 bytes = 3.00 GiB
131,072 tokens: 12,884,901,888 bytes = 12.00 GiB
262,144 tokens: 25,769,803,776 bytes = 24.00 GiB
Each token of context costs 96 KiB, and one request that fills the model's 262,144-token limit would hold 24 GiB of cache by itself.
The file also shows why the script reads head_dim directly: dividing this model's hidden size of 2,048 by its 32 attention heads gives 64, half the declared value.
Three limits apply to these numbers:
- The formula is assembled, not quoted. The paper's version has no KV-head term, and vLLM's code gives the size for one layer, so no single source states the product we used.
- The 2 bytes per value is the width of the data type: PyTorch lists
torch.bfloat16as a 16-bit floating point type. vLLM's--kv-cache-dtypedefaults to the model's data type and can be set to others, which changes that term. - The result is an estimate. We did not compare it with the cache size a running server prints in its log, and it ignores the partly filled last block of each sequence.
Sources: our script, run with Python 3.12.16; Qwen, config.json at commit b2cff646eb4bb1d68355c01b18ae02e7cf42d120; PyTorch 2.15 documentation, Tensor Attributes; vLLM v0.33.0 documentation, Engine Arguments.
The problem PagedAttention solves
A cache that large and that variable is hard to store, which is the problem vLLM was built around. The paper's abstract describes it: "the key-value cache (KV cache) memory for each request is huge and grows and shrinks dynamically." Earlier systems kept each request's cache "in contiguous memory space", reserving a chunk sized for the request's maximum length, and the paper names three kinds of waste that result:
- "reserved slots for future tokens";
- "internal fragmentation due to over-provisioning for potential maximum sequence lengths";
- "external fragmentation from the memory allocator".
PagedAttention removes the need for one contiguous region. It "partitions the KV cache of each sequence into KV blocks. Each block contains the key and value vectors for a fixed number of tokens", and a block table records "the mapping between logical and physical KV blocks of each request." The authors borrow the vocabulary of an operating system: "one can think of blocks as pages, tokens as bytes, and requests as processes." Blocks are allocated as a sequence grows, and because they are all the same size, the design "eliminates external fragmentation."
In vLLM the block size is the --block-size argument, documented as the "Size of a contiguous cache block in number of tokens."
Sources: Kwon et al., PagedAttention, abstract and sections 1, 3.1, 4.1 and 4.2; vLLM v0.33.0 documentation, Engine Arguments.
How vLLM sizes the cache
By default vLLM pre-allocates the cache, so its size is set through a few arguments.
--gpu-memory-utilizationis "The fraction of GPU memory to be used for the model executor, which can range from 0 to 1." The v0.33.0 documentation gives its default as 0.92. By default vLLM will "automatically infer the kv cache size based on gpu_memory_utilization", and the tuning guide adds that "vLLM pre-allocates GPU cache using this percentage of memory. By increasing utilization, you can provide more KV cache space."--kv-cache-memory-bytessets the "Size of KV Cache per GPU in bytes" directly, and when it is set it "ignores gpu_memory_utilization."--max-model-lenis the "Model context length (prompt and output)", taken from the model configuration when unset. The valueautoasks vLLM to "choose the maximum model length that fits in GPU memory."--max-num-seqsis the "Maximum number of sequences to be processed in a single iteration." The memory guide lists it withmax_model_lenas the two options that "further reduce memory usage."
The server reports the outcome at startup.
Its log has a GPU KV cache size line, which "reports the total number of tokens that can be stored in the GPU KV cache at once", and a Maximum concurrency line that divides that capacity by max_model_len.
With our estimate, a capacity in tokens converts to bytes at 96 KiB each for this model, which is the comparison we would make on a running server.
When the cache fills anyway, vLLM "can preempt requests to free up KV cache space for other requests. Preempted requests are recomputed when sufficient KV cache space becomes available again." Planning memory for inference on one DGX Spark places this cache in the memory budget of a whole machine.
Sources: vLLM v0.33.0 documentation, Engine Arguments, Optimization and Tuning, Conserving Memory and Parallelism and Scaling.
Prefix caching
Because a block's contents depend only on the tokens up to it, two requests that start the same way can share blocks. vLLM calls this automatic prefix caching: it "caches the KV cache of existing queries, so that a new query can directly reuse the KV cache if it shares the same prefix with one of the existing queries, allowing the new query to skip the computation of the shared part."
The design page gives the mechanics:
- Identity: "we hash each kv-cache block by the tokens in the block and the tokens in the prefix before the block."
- Granularity: "We only cache full blocks." In the page's example with 4-token blocks, a request sharing 10 tokens with an earlier one reuses 2 blocks, 8 tokens.
- Lifetime: blocks of finished requests stay reusable until the space is needed, and the least recently used block is evicted first.

- One contiguous chunk: earlier systems kept the cache of each request in contiguous memory space, reserving a chunk sized for the maximum length of the request; the paper counts reserved slots for future tokens among the waste.
- KV blocks: PagedAttention partitions the cache of each sequence into KV blocks, allocated as a sequence grows and all the same size.
- Two requests share blocks: because the contents of a block depend only on the tokens up to it, two requests that start the same way can share blocks.
- Only full blocks are cached: in the example on the design page, with 4-token blocks, a request sharing 10 tokens with an earlier one reuses 2 blocks, 8 tokens.
The documentation names two workloads that fit, repeated questions about one long document and a multi-round conversation, and one limit: prefix caching "only reduces the time of processing the queries (the prefilling phase) and does not reduce the time of generating new tokens (the decoding phase)."
A coding agent that resends a growing conversation on every turn is the second workload.
The switch is --enable-prefix-caching or --disable-prefix-caching; check which one your version starts with before relying on it.
Sources: vLLM v0.33.0 documentation, Automatic Prefix Caching, Automatic Prefix Caching design and Engine Arguments.
What an attacker targets here
Sharing blocks between requests makes one request's speed depend on what another request sent, and vLLM has published an advisory about exactly that.
The advisory is GHSA-4qjh-9fv9-r85r, CVE-2025-46570, titled "Potential Timing Side-Channel Vulnerability in vLLM's Chunk-Based Prefix Caching" and published on May 28, 2025. It says:
- When a new prompt matches a cached prefix chunk, "the prefill process speeds up, which is reflected in the TTFT (Time to First Token)."
- The reporters found those timing differences "significant enough to be recognized and exploited."
- If a victim's prompt or a system prompt is cached, "an attacker sharing the same backend could attempt to guess the victim's input. By measuring the TTFT based on prefix matches, the attacker could verify if their guess is correct, leading to potential leakage of private information."
The advisory rates the severity Low, with a CVSS score of 2.6 and a vector that lists high attack complexity and low privileges required. It lists versions before 0.9.0 as affected and 0.9.0 as patched, and the reporters tested on vLLM v0.5.1.
The condition in that text is an attacker "sharing the same backend", so the question to ask of any deployment is who else can send requests to the same server. The control that limits that set is authentication in front of anything that must be reachable from another machine, one of the controls Model servers are web apps lists for our own servers.
Source: vLLM security advisory GHSA-4qjh-9fv9-r85r.
Where to go next
When one machine cannot hold the weights and the cache together, one option is to split the model across machines, and Tensors and tensor parallelism explains how that split works and what it asks of the network. When eight-way tensor parallelism is worth testing covers what happens to the KV heads when the split is wider than their count.