The KV Cache Is the Real Tenant of Your GPU

When people ask me whether a model "fits" on a GPU, they're usually thinking about the weights. Mistral 7B in FP16 is about 14.5 GB, so yes, easily.

What actually fills the GPU is something else: the KV cache.

A quick explanation for anyone who hasn't run into it. An LLM writes one token at a time, and each new token has to look back at everything before it. To avoid recomputing the whole conversation at every step, the model keeps intermediate values (keys and values) for every past token, at every layer. That's what makes generation fast. It also lives in GPU memory, per user, for as long as the request runs.

For Mistral 7B it's roughly 128 KB per token. A 4,000-token conversation takes half a gigabyte. A hundred people chatting at once is around 50 GB, more than three times the model itself. On a 120B model with a 6,000-token RAG prompt, it's about 2 GB per request.

So the number of users one GPU can serve is roughly its free memory divided by the cache each request needs, and that cache grows with context length.

This is why I get a little nervous when a team says "let's send the top 20 chunks" or "the model supports 128K of context, why not use it". Those are capacity decisions, even if nobody meant them that way. Same for agents that send 40 tool definitions on every call.

What helps: prefix caching, since system prompts and tool definitions repeat constantly. Storing the cache in FP8, which halves it. Measuring context length at p95 rather than on average. And better retrieval. Five good chunks beat twenty mediocre ones on accuracy and on cost.

Does anyone on your team actually own context length?

#LLMOps #RAG #GPU