KV Cache

Serving LLMs is as much a memory problem as a compute problem.

Here is why:

𝟭. 𝗟𝗟𝗠𝘀 𝗴𝗲𝗻𝗲𝗿𝗮𝘁𝗲 𝗼𝗻𝗲 𝘁𝗼𝗸𝗲𝗻 𝗮𝘁 𝗮 𝘁𝗶𝗺𝗲

An LLM does not produce a full answer in one pass.

It predicts the next token, appends it, then runs again.

So a 500-token response means 500 forward passes through the model.

That is why latency climbs so quickly.

𝟮. 𝗧𝗵𝗲 𝗞𝗩 𝗰𝗮𝗰𝗵𝗲 𝗶𝘀 𝘁𝗵𝗲 𝗺𝗲𝗺𝗼𝗿𝘆 𝗺𝗼𝗻𝘀𝘁𝗲𝗿

To generate the next token, the model needs the Key and Value vectors from every previous token.

Those vectors are stored in GPU memory as the KV cache.

On Llama 3 70B, that works out to roughly 320 KB per token in FP16.

So:

• 2,000 tokens is about 640 MB
• 32,000 tokens is about 10 GB
• 128,000 tokens is about 40 GB for one user

That is why long-context serving gets expensive fast.

𝟯. 𝗚𝗣𝗨𝘀 𝗮𝗿𝗲 𝗹𝗶𝗺𝗶𝘁𝗲𝗱 𝗯𝘆 𝗯𝗮𝗻𝗱𝘄𝗶𝗱𝘁𝗵, 𝗻𝗼𝘁 𝗷𝘂𝘀𝘁 𝗰𝗼𝗺𝗽𝘂𝘁𝗲

A GPU has different memory layers:

• CPU memory: large, but slow
• HBM / VRAM: fast, but limited
• SRAM: extremely fast, but tiny

During inference, the GPU keeps moving weights and KV cache data from HBM into SRAM so the tensor cores can do the maths.

That means performance is often limited by how fast data can move, not how fast the compute units are.

𝗧𝗵𝗲 𝗿𝗲𝗮𝗹 𝗹𝗲𝘀𝘀𝗼𝗻

Most LLM inference optimisations do one of two things:

• move less data
• use GPU memory more efficiently

That is why quantisation helps.
That is why PagedAttention helps.