What Is Key-Value (KV) Caching in LLMs and How Does It Speed Up Inference?

The essay-rewriting analogy

To understand What Is Key-Value (KV) Caching in LLMs and How Does It Speed Up Inference?, we must first look at how Large Language Models generate text. An LLM produces text auto-regressively, meaning it generates text one word—or token—at a time. Each new token generated is appended to the prompt, and the entire sequence is fed back into the model to calculate the next token. Without optimization, this process is incredibly inefficient.

Imagine a human writing an essay under a bizarre rule: before writing each new word, they must re-read the entire essay from the very beginning, re-analyzing the grammatical structures, context, and semantic relationships of every word they have already written. By the time they reach the 500th word, they are spending massive mental energy re-processing the first 499 words just to figure out word number 500. This is exactly how a standard Transformer model operates without a cache.

KV caching acts as the model's scratchpad. When the model processes the initial prompt and generates the first few tokens, it calculates internal mathematical representations—specifically called "Key" and "Value" vectors—for every single token. Instead of discarding these vectors and recalculating them from scratch for the next token, the engine saves them in high-speed GPU memory (VRAM). When calculating the next token, the model only computes the Key and Value vectors for the single newly generated token and appends them to the saved cache. By preserving this computational history, KV caching removes the repeated Key and Value projections for earlier tokens, so each new token costs far less than it would if the whole prefix were reprocessed.

Prefill, decoding, and the memory formula

To understand the mechanics of KV caching, we have to look under the hood at the Self-Attention mechanism of the Transformer architecture. Self-attention determines how much focus a model should place on other parts of the input sequence when processing a specific token. This is accomplished using three projection matrices: Queries ($Q$), Keys ($K$), and Values ($V$).

For a given token, the model projects its embedding into these three vectors:

1. Query ($Q$): Represents "what I am looking for" in the rest of the text.
2. Key ($K$): Represents "what context I contain" to match against queries.
3. Value ($V$): Contains the actual semantic information that gets aggregated into the output if the Key matches a Query.

The attention calculation is mathematically defined as:

Attention(Q, K, V) = softmax( (Q * Kᵀ) / √d_k ) * V

During the initial "prefill" phase, the LLM processes the entire user prompt. It computes the $Q$, $K$, and $V$ matrices for every input token. This is a highly parallelized operation where the GPU can utilize its massive compute capacity efficiently.

However, during the subsequent "decoding" phase, the model generates tokens one by one. For token $i$, the model only needs the Query vector $Q_i$ to see how it matches with all previous tokens. To compute the attention output, $Q_i$ must be multiplied by the Keys ($K_{1..i}$) and Values ($V_{1..i}$) of all previous tokens. If we do not cache $K_{1..i-1}$ and $V_{1..i-1}$, the GPU is forced to recompute these vectors for every single past token, over and over, for every step. This redundant calculation is a massive waste of floating-point operations (FLOPs).

By implementing a KV cache, the system stores $K_{1..i-1}$ and $V_{1..i-1}$ in memory. During decoding step $i$, the model only projects the new token to get $Q_i$, $K_i$, and $V_i$. It then appends $K_i$ and $V_i$ to the existing cache, computes the attention product using the stored values, and produces the next token. This removes the repeated K/V projection work for all previous tokens, so that part of each step no longer grows with sequence length. The attention step still reads all $N$ cached entries, so its cost per token remains linear in $N$.

The Memory Footprint of KV Cache

The physical size of the KV cache can be staggering, especially when handling long sequences. The size of the KV cache (in bytes) for a single inference request can be calculated with the following formula:

Cache Size = 2 * Real_Tokens * Layers * Heads * Head_Dim * Precision_Bytes

Where the factor of 2 accounts for storing both Keys and Values, "Layers" is the number of transformer layers, "Heads" is the number of attention heads, "Head_Dim" is the dimension of each head, and "Precision_Bytes" is the numeric precision (e.g., 2 bytes for FP16 or BF16, 1 byte for INT8 quantization).

For example, if you run a batch of 16 requests on a mid-sized model with 32 layers, 32 attention heads, a head dimension of 128, and a sequence length of 4,096 tokens in 16-bit precision, the KV cache alone demands roughly 34 GB of GPU VRAM (about 2.1 GB per request, times 16 requests). When scaling to modern context limits, such as the expansive capabilities of Google's Gemini 3.6 Flash or Anthropic's Claude Sonnet 5, managing this memory footprint becomes the primary challenge of AI engineering.

PagedAttention, GQA/MQA, and prompt caching

To make KV caching practical at scale, software engineers and hardware architects have developed clever optimizations. These technologies allow enterprise systems to support thousands of concurrent users without running out of high-bandwidth memory.

1. PagedAttention (vLLM)

In early LLM serving frameworks, KV caches had to be stored in contiguous virtual memory blocks. Because sequence lengths are dynamic and unpredictable, systems had to pre-allocate memory for the maximum possible sequence length (e.g., reserving space for a full 8,000-token context window upfront). This led to massive memory fragmentation and waste, The vLLM paper measured that only 20.4%–38.2% of KV cache memory held actual token states in the systems it studied, meaning roughly 62%–80% was wasted.

The development of PagedAttention (introduced by the vLLM project) solved this by borrowing virtual memory paging concepts from traditional operating systems. PagedAttention partitions the KV cache of each request into small, non-contiguous physical blocks. These blocks are allocated dynamically as tokens are generated, greatly reducing fragmentation, and the paper reports 2–4× higher throughput than FasterTransformer and Orca at the same latency, which lowers the cost of serving requests.

2. Multi-Query Attention (MQA) and Grouped-Query Attention (GQA)

To reduce the sheer physical size of the KV cache, model architectures have shifted away from Multi-Head Attention (MHA), where every attention query head has its own distinct Key and Value heads. Instead, modern models employ:

• Multi-Query Attention (MQA): All Query heads share a single Key and Value head. This shrinks the KV cache by a factor equal to the number of query heads (for example 32× for a 32-head model) but can reduce model quality.
• Grouped-Query Attention (GQA): A highly effective middle ground where Query heads are grouped (e.g., 8 Query heads per 1 Key/Value head). The GQA paper reports that uptrained GQA reaches quality close to multi-head attention with speed comparable to MQA, and the cache shrinks by the grouping factor (8 query heads per Key/Value head gives an 8× reduction). GQA is standard in models like Meta's Llama 3 series and Mistral models.

3. Prompt Caching for Agentic Workflows

In highly interactive scenarios—such as multi-turn conversations with OpenAI's GPT-5.6 Sol or complex developer tasks executed by Claude Opus 5—users repeatedly send a highly overlapping system context, historical chat log, or source code file. In these scenarios, the system prompt and history can scale to tens of thousands of AI tokens.

API providers leverage KV caching globally to enable Prompt Caching. When a user submits a prompt, the hosting engine checks if the prefix of that prompt matches an already processed request. If it matches, the engine reuses the stored KV state for that prefix and skips the prefill computation for it. This reduces Time-to-First-Token for long prompts, and providers bill cached reads at a fraction of the normal input price; the exact multiplier differs by provider and model, so read it from the provider's pricing page.

KV cache compared with prompt caching, vector indexes, and weights

To prevent confusion, it is helpful to distinguish KV caching from other common memory, database, and caching concepts in the AI field. Engineers often mix these terms up when discussing performance optimization.

By comparing these concepts, we can see that while a vector database index helps an AI agent pull relevant information from a vast database, KV caching is the immediate, real-time memory mechanism that allows the model's core attention engine to process that retrieved data without grinding to a halt.

Three myths about KV caching

Because KV caching operates quietly in the background of hosting frameworks, several myths persist among developers and product managers regarding its behavior and limitations.

Misconception 1: "KV Caching Is Free and Has No Drawbacks"

While KV caching cuts the work done per token, it does so by trading computation for memory. As sequence lengths and concurrent user numbers increase, the KV cache can easily consume all available VRAM, leading to "Out of Memory" (OOM) errors. For enterprise deployments, managing the extreme VRAM pressure of KV caches is often the costliest engineering challenge, requiring compromises like quantizing the cache to 8-bit or 4-bit precision or deploying complex routing strategies.

Misconception 2: "KV Caching Makes the Model More Intelligent"

KV caching does not alter the mathematical output of the model in any way. A model running with KV caching will output the exact same tokens as a model running without it. It is purely a hardware-level execution speedup. The internal weights of the model are untouched, and the logical accuracy of the generations remains identical.

Misconception 3: "Every LLM Implementation Handles KV Caching the Same Way"

How KV caching is handled depends heavily on the model's architectural blueprint and the inference engine. Some older open-source models do not support GQA, resulting in massive, inefficient KV caches. Additionally, advanced serving stacks use specialized optimizations like RadixAttention or deep page-swapping mechanics to maximize resource reuse, while basic custom scripts might use naive, unoptimized caching strategies that waste VRAM.

The trade-off to remember

Understanding What Is Key-Value (KV) Caching in LLMs and How Does It Speed Up Inference? is essential for anyone designing, deploying, or hosting modern generative AI applications. By saving the intermediate Key and Value representations of previously processed tokens, KV caching prevents massive, redundant matrix multiplications, turning repeated recomputation into incremental work at each step.

While it introduces major memory bandwidth and capacity challenges, techniques like Grouped-Query Attention (GQA) and software breakthroughs like PagedAttention have made modern, long-context LLMs and real-time agentic systems commercially viable. As models scale up in context and reasoning capabilities, mastering KV cache optimization remains one of the most critical frontiers in AI systems engineering.

Counting the work a KV cache saves

The toy decoder below is one attention head with a 64-dimensional state, written in NumPy 1.26.4 on Python 3.10.0. It generates 600 steps twice with identical weights and inputs: once re-projecting Keys and Values for the whole prefix at every step, and once keeping them in a cache.

Script

"""Toy single-head attention decoder: recompute K/V every step vs. keep a KV cache.
Same weights, same inputs; checks the outputs are identical and times both."""
import time
import numpy as np

rng = np.random.default_rng(0)
d, steps = 64, 600
Wq, Wk, Wv = (rng.standard_normal((d, d)) / np.sqrt(d) for _ in range(3))
x0 = rng.standard_normal(d)

def attend(q, K, V):
    s = K @ q / np.sqrt(d)
    w = np.exp(s - s.max())
    w /= w.sum()
    return w @ V

def run(use_cache):
    xs = [x0]
    kv_rows = 0
    Kc, Vc = [], []
    t0 = time.perf_counter()
    for _ in range(steps):
        if use_cache:
            Kc.append(Wk @ xs[-1])
            Vc.append(Wv @ xs[-1])
            K, V = np.array(Kc), np.array(Vc)
            kv_rows += 1
        else:
            X = np.array(xs)
            K, V = X @ Wk.T, X @ Wv.T
            kv_rows += len(xs)
        out = attend(Wq @ xs[-1], K, V)
        xs.append(np.tanh(out))
    return np.array(xs), kv_rows, time.perf_counter() - t0

a, rows_a, t_a = run(False)
b, rows_b, t_b = run(True)
print(f"numpy {np.__version__}, d={d}, steps={steps}")
print(f"outputs identical (allclose): {np.allclose(a, b)}   max abs diff: {np.abs(a - b).max():.2e}")
print(f"K/V rows projected  no-cache: {rows_a}   cache: {rows_b}   ratio: {rows_a / rows_b:.0f}x")
print(f"wall time           no-cache: {t_a * 1000:.0f} ms   cache: {t_b * 1000:.0f} ms")
bytes_per_step = 2 * d * 8
print(f"cache memory after {steps} steps: {steps * bytes_per_step / 1024:.0f} KiB (float64, 1 head, 1 layer)")

Output

numpy 1.26.4, d=64, steps=600
outputs identical (allclose): True   max abs diff: 4.44e-16
K/V rows projected  no-cache: 180300   cache: 600   ratio: 300x
wall time           no-cache: 743 ms   cache: 292 ms
cache memory after 600 steps: 600 KiB (float64, 1 head, 1 layer)

The two runs produced the same outputs (largest difference 4.4e-16, floating-point noise). Without a cache the decoder projected 180,300 Key/Value rows, the sum of 1 to 600; with a cache it projected 600, about 300 times fewer, which is what the arithmetic predicts. Wall-clock time improved by a smaller factor, 743 ms against 292 ms, because at this size the Python loop and the attention pass dominate. Read the timing as a property of this toy, not as a speedup figure for any real model.

The last line shows the cost side: the cache grows by one Key and one Value vector per step, so memory is linear in sequence length. A real model multiplies that by its layers, heads and batch size.

This run did not involve a real LLM, a GPU or a serving stack such as vLLM, so it says nothing about their memory-bandwidth behaviour.

Sources and Further Reading

• Cache strategies — Hugging Face
• Efficient Memory Management for Large Language Model Serving with PagedAttention — arXiv
• Prompt caching — Claude Docs
• Pricing — Claude Docs
• GQA: Training Generalized Multi-Query Transformer Models from Multi-Head Checkpoints — arXiv

Memory optimization flags and default configurations in vLLM can change rapidly, so verify current parameters against the official documentation.