The KV cache is memory a model server spends on every token of every active request, and four config values set its size per token. It is why a machine that holds a model's weights can still run short of memory for a long prompt or a second user. We explain what the cache stores, estimate its size for one public model with a short script, and go through the vLLM settings that budget it and reuse it.

What the KV cache stores

The cache exists because a language model writes one token at a time and each new token needs all the earlier ones. The paper that introduced vLLM puts it this way: "the generation process of each new token depends on all the previous tokens in that sequence, specifically their key and value vectors." Those vectors come from the attention layers, and the paper notes that "each token has a set of key and value vectors across layers and attention heads within a layer."

Without a cache, the model would rebuild them for the whole sequence at every step. The Hugging Face Transformers documentation describes the waste and the fix: the model "performs the same computations each time", and "A KV cache stores these calculations so they can be reused without recomputing them." With the cache in place, each step computes keys and values for the newest token only and reads the rest from memory.

One property matters for everything that follows. The paper states that "the KV cache of one token depends on all its previous tokens", so the same word at a different position, or after a different prefix, has a different entry.

Sources: Kwon et al., Efficient Memory Management for Large Language Model Serving with PagedAttention, arXiv 2309.06180, sections 2.2 and 4.1; Hugging Face Transformers documentation, Cache strategies.

What its size scales with

Since every token keeps vectors in every layer, the cache grows along several axes at once. The paper gives the arithmetic for its example, a 13B-parameter OPT model: "the KV cache of a single token demands 800 KB of space, calculated as 2 (key and value vectors) × 5120 (hidden state size) × 40 (number of layers) × 2 (bytes per FP16)." It then scales that up twice:

Newer models change one term. Grouped-query attention lets several query heads share one key and value head, and the paper that proposed it points out that fewer key and value heads mean a smaller cache: going to a single shared head reduces "the size of the key-value cache and therefore amount of data that needs to be loaded by a factor of H", the number of heads. A model's configuration file reports the count as num_key_value_heads, which the Transformers documentation describes as "the number of key_value heads that should be used to implement Grouped Query Attention."

vLLM's source shows the same terms from the implementation side. For one attention layer, a cache page holds by default, for each KV head and each token in the block, (self.head_size + self.head_size_v) * get_dtype_size(self.dtype) bytes: a key and a value of one head's width. Its Llama model code reads the KV head count from num_key_value_heads, falling back to num_attention_heads, and reads the head width from head_dim when the configuration has one, otherwise dividing the hidden size by the attention head count.

That gives five factors: layers, KV heads, head width, bytes per value, and tokens held.

Sources: Kwon et al., PagedAttention, section 3; Ainslie et al., GQA: Training Generalized Multi-Query Transformer Models from Multi-Head Checkpoints, arXiv 2305.13245, section 2.2; Hugging Face Transformers documentation, Qwen3MoE; vLLM v0.33.0 source, kv_cache_interface.py and llama.py, the head width read and the KV head count read.

Estimating the size for one model

Those five factors are enough for an estimate from public numbers, before any hardware is involved. We use Qwen/Qwen3-Coder-30B-A3B-Instruct, whose config.json declares 48 layers, 4 KV heads, a head_dim of 128 and the bfloat16 data type. We downloaded the file at a pinned commit, so the numbers cannot move under us:

curl -sSo config.json https://huggingface.co/Qwen/Qwen3-Coder-30B-A3B-Instruct/raw/b2cff646eb4bb1d68355c01b18ae02e7cf42d120/config.json

The script multiplies the factors and prints the result at a few context lengths:

"""Our estimate of KV cache size from a model's config.json (not vLLM's allocator)."""
import json
import sys

BYTES_PER_VALUE = {"bfloat16": 2, "float16": 2, "float32": 4}

with open(sys.argv[1], encoding="utf-8") as f:
    cfg = json.load(f)

layers = cfg["num_hidden_layers"]
kv_heads = cfg.get("num_key_value_heads", cfg["num_attention_heads"])
head_dim = cfg.get("head_dim") or cfg["hidden_size"] // cfg["num_attention_heads"]
width = BYTES_PER_VALUE[cfg["torch_dtype"]]

per_token = 2 * layers * kv_heads * head_dim * width  # 2 = one key and one value
print(f"layers={layers} kv_heads={kv_heads} head_dim={head_dim} dtype={cfg['torch_dtype']} ({width} bytes)")
print(f"bytes per token: {per_token:,} ({per_token // 1024} KiB)")
for tokens in (8_192, 32_768, 131_072, cfg["max_position_embeddings"]):
    size = per_token * tokens
    print(f"{tokens:>7,} tokens: {size:>14,} bytes = {size / 2**30:5.2f} GiB")

We ran it in a throwaway container with no network and a read-only filesystem:

$ docker run --rm --network none --read-only -v "$PWD":/demo:ro -w /demo python:3.12-slim python kv_cache_size.py config.json
layers=48 kv_heads=4 head_dim=128 dtype=bfloat16 (2 bytes)
bytes per token: 98,304 (96 KiB)
  8,192 tokens:    805,306,368 bytes =  0.75 GiB
 32,768 tokens:  3,221,225,472 bytes =  3.00 GiB
131,072 tokens: 12,884,901,888 bytes = 12.00 GiB
262,144 tokens: 25,769,803,776 bytes = 24.00 GiB

Each token of context costs 96 KiB, and one request that fills the model's 262,144-token limit would hold 24 GiB of cache by itself. The file also shows why the script reads head_dim directly: dividing this model's hidden size of 2,048 by its 32 attention heads gives 64, half the declared value.

Three limits apply to these numbers:

Sources: our script, run with Python 3.12.16; Qwen, config.json at commit b2cff646eb4bb1d68355c01b18ae02e7cf42d120; PyTorch 2.15 documentation, Tensor Attributes; vLLM v0.33.0 documentation, Engine Arguments.

The problem PagedAttention solves

A cache that large and that variable is hard to store, which is the problem vLLM was built around. The paper's abstract describes it: "the key-value cache (KV cache) memory for each request is huge and grows and shrinks dynamically." Earlier systems kept each request's cache "in contiguous memory space", reserving a chunk sized for the request's maximum length, and the paper names three kinds of waste that result:

PagedAttention removes the need for one contiguous region. It "partitions the KV cache of each sequence into KV blocks. Each block contains the key and value vectors for a fixed number of tokens", and a block table records "the mapping between logical and physical KV blocks of each request." The authors borrow the vocabulary of an operating system: "one can think of blocks as pages, tokens as bytes, and requests as processes." Blocks are allocated as a sequence grows, and because they are all the same size, the design "eliminates external fragmentation."

In vLLM the block size is the --block-size argument, documented as the "Size of a contiguous cache block in number of tokens."

Sources: Kwon et al., PagedAttention, abstract and sections 1, 3.1, 4.1 and 4.2; vLLM v0.33.0 documentation, Engine Arguments.

How vLLM sizes the cache

By default vLLM pre-allocates the cache, so its size is set through a few arguments.

The server reports the outcome at startup. Its log has a GPU KV cache size line, which "reports the total number of tokens that can be stored in the GPU KV cache at once", and a Maximum concurrency line that divides that capacity by max_model_len. With our estimate, a capacity in tokens converts to bytes at 96 KiB each for this model, which is the comparison we would make on a running server.

When the cache fills anyway, vLLM "can preempt requests to free up KV cache space for other requests. Preempted requests are recomputed when sufficient KV cache space becomes available again." Planning memory for inference on one DGX Spark places this cache in the memory budget of a whole machine.

Sources: vLLM v0.33.0 documentation, Engine Arguments, Optimization and Tuning, Conserving Memory and Parallelism and Scaling.

Prefix caching

Because a block's contents depend only on the tokens up to it, two requests that start the same way can share blocks. vLLM calls this automatic prefix caching: it "caches the KV cache of existing queries, so that a new query can directly reuse the KV cache if it shares the same prefix with one of the existing queries, allowing the new query to skip the computation of the shared part."

The design page gives the mechanics:

Three panels on one platform, the KV cache memory. Earlier systems, one chunk: the request's cache and reserved slots for future tokens. One request in KV blocks: two 4-token blocks and the next block, added as it grows. A new request with the same start: block 1 reused, block 2 reused, next block not reused. Lanes: 1 same request, stored in blocks; 2 full blocks reused.
  1. One contiguous chunk: earlier systems kept the cache of each request in contiguous memory space, reserving a chunk sized for the maximum length of the request; the paper counts reserved slots for future tokens among the waste.
  2. KV blocks: PagedAttention partitions the cache of each sequence into KV blocks, allocated as a sequence grows and all the same size.
  3. Two requests share blocks: because the contents of a block depend only on the tokens up to it, two requests that start the same way can share blocks.
  4. Only full blocks are cached: in the example on the design page, with 4-token blocks, a request sharing 10 tokens with an earlier one reuses 2 blocks, 8 tokens.
The outlines are not to scale. The 4-token blocks and the 10 shared tokens are the example on the vLLM design page, not a measurement of a model.

The documentation names two workloads that fit, repeated questions about one long document and a multi-round conversation, and one limit: prefix caching "only reduces the time of processing the queries (the prefilling phase) and does not reduce the time of generating new tokens (the decoding phase)." A coding agent that resends a growing conversation on every turn is the second workload. The switch is --enable-prefix-caching or --disable-prefix-caching; check which one your version starts with before relying on it.

Sources: vLLM v0.33.0 documentation, Automatic Prefix Caching, Automatic Prefix Caching design and Engine Arguments.

What an attacker targets here

Sharing blocks between requests makes one request's speed depend on what another request sent, and vLLM has published an advisory about exactly that.

The advisory is GHSA-4qjh-9fv9-r85r, CVE-2025-46570, titled "Potential Timing Side-Channel Vulnerability in vLLM's Chunk-Based Prefix Caching" and published on May 28, 2025. It says:

The advisory rates the severity Low, with a CVSS score of 2.6 and a vector that lists high attack complexity and low privileges required. It lists versions before 0.9.0 as affected and 0.9.0 as patched, and the reporters tested on vLLM v0.5.1.

The condition in that text is an attacker "sharing the same backend", so the question to ask of any deployment is who else can send requests to the same server. The control that limits that set is authentication in front of anything that must be reachable from another machine, one of the controls Model servers are web apps lists for our own servers.

Source: vLLM security advisory GHSA-4qjh-9fv9-r85r.

Where to go next

When one machine cannot hold the weights and the cache together, one option is to split the model across machines, and Tensors and tensor parallelism explains how that split works and what it asks of the network. When eight-way tensor parallelism is worth testing covers what happens to the KV heads when the split is wider than their count.