Why KV Cache Stores K and V Vectors But Never Q?

The Unseen Hero Behind Fast and Efficient LLM Inference

When you ask an AI to generate text, there’s a lot happening behind the scenes. One of the most critical optimizations that keeps modern language models running fast is something called KV caching. But here’s the puzzle: why do we cache K and V vectors, but never Q? Let’s dive deep into understanding this elegant engineering choice.

Understanding the Attention Mechanism

At the heart of every large language model lies the Attention mechanism. The formula is deceptively simple:

Attention(Q, K, V) = softmax(QK^T / √d_k) V

Here’s what each component represents:

• Q (Query): Represents the current token asking “what information do I need?”
• K (Key): The keys from all previous tokens that help identify relevant information
• V (Value): The actual information content associated with each token

To understand why caching works the way it does, we need to think about how token generation happens.

The Token Generation Process: Autoregressive Generation

Language models generate text one token at a time in an autoregressive manner. Here’s what happens:

1. Token 1 (“The”) is generated → compute Q₁, K₁, V₁
2. Token 2 (“cat”) is generated → compute Q₂, and it attends to K₁, K₂ and V₁, V₂
3. Token 3 (“sat”) is generated → compute Q₃, and it attends to K₁, K₂, K₃ and V₁, V₂, V₃
4. Token 4 (“on”) is generated → compute Q₄, and it attends to K₁, K₂, K₃, K₄ and V₁, V₂, V₃, V₄
5. And so on…

See the pattern? Each new token needs to attend to all previous keys and values, but each token has a unique query.

Why Cache K?

Every future token needs past Keys.

• When we generate the 5th token, we must attend to keys from tokens 1, 2, 3, and 4
• When we generate the 100th token, we must attend to keys from all 99 previous tokens
• Without caching: we’d recompute K₁, K₂, K₃… K₉₉ from scratch every single time
• With caching: we store them once and reuse them, avoiding expensive recomputation

Benefit: Caching K vectors saves tremendous computation. Computing expensive matrix operations repeatedly would be wasteful when the answer never changes.

Why Cache V?

Values hold the actual information needed forever.

The Key-Value pairs form an information retrieval system. Once we compute the value representation of a token, that representation remains relevant and useful for all future tokens.

• Token 5 needs V₁ through V₄ to learn what context is available
• Token 50 still needs those same V₁ through V₄—they haven’t changed
• Token 1000 still needs them

Benefits:

• Values encode the semantic content of each token
• Every future token needs this information to make informed predictions
• By caching, we avoid recomputing these valuable representations
• This saves massive computational overhead

Why NOT Cache Q?

Queries are unique and ephemeral they’re only used once.

This is the key insight that makes the design elegant:

1. Each token has a unique query: When you generate token 5, you create Q₅ specifically for that token’s needs. It’s never computed again.
2. Query is used only in the current step: Q₅ is multiplied with K₁…K₅ to produce attention weights, but then it’s discarded. It won’t be needed when generating token 6.
3. Caching would be wasted memory: If we cached all queries, we’d be storing:
   • Q₁ (never used again after token 1)
   • Q₂ (never used again after token 2)
   • Q₃ (never used again after token 3)
   • … Q₉₉ (never used again after token 99)
4. Zero reuse = zero benefit: Queries are fundamentally different—each token “asks” different questions based on its unique position and context.

The principle: “Caching Q = wasted memory, zero reuse”

The Memory and Speed Tradeoff

Without KV caching, generating a sequence of 100 tokens would require:

• 100 forward passes through the attention layer
• Recomputing K and V vectors 100 times each
• Quadratic complexity in sequence length

With KV caching:

• We compute K and V once per new token
• Reuse all previous K and V values
• Linear complexity in sequence length
• Massive speedup: Often 10-100x faster for long sequences

This is why modern LLMs can generate tokens so quickly. Every inference call doesn’t have to re-process the entire context.

Autoregressive Generation in Action

Here’s how it plays out in practice:

Token: The        cat        sat        on        the        mat
       ↓          ↓          ↓          ↓         ↓          ↓
Q:     Q₁(fresh)  Q₂(fresh)  Q₃(fresh)  Q₄(fresh) Q₅(fresh)  Q₆(fresh)
K:     K₁        K₁, K₂     K₁, K₂, K₃ K₁...K₄   K₁...K₅   K₁...K₆
V:     V₁        V₁, V₂     V₁, V₂, V₃ V₁...V₄   V₁...V₅   V₁...V₆

Notice:

• Q is always fresh for each new token (never cached, never reused)
• K is accumulated in the cache with each new token
• V is accumulated in the cache with each new token

We don’t cache Q because it’s a question asked once.
We cache K and V because they’re answers needed again and again.

This design choice reflects a fundamental insight about how attention works:

• Queries are ephemeral – they’re specific to one generation step
• Keys and Values are persistent – they’re needed by all future steps

By caching selectively, we get the best of both worlds:

This is why KV caching has become the standard optimization in modern LLM inference, enabling the fast, responsive AI assistants we use every day.