vLLM

LLM inference looks complicated.
It mostly comes down to 4 runtime tricks.

𝟭) 𝗖𝗼𝗻𝘁𝗶𝗻𝘂𝗼𝘂𝘀 𝗯𝗮𝘁𝗰𝗵𝗶𝗻𝗴
Old batching waits for the slowest request.

Continuous batching swaps in a new request the moment another one finishes.

That keeps the GPU busy instead of leaving slots idle.

𝟮) 𝗣𝗮𝗴𝗲𝗱𝗔𝘁𝘁𝗲𝗻𝘁𝗶𝗼𝗻
KV cache is the memory bottleneck.

Instead of reserving one huge contiguous block, PagedAttention splits it into small pages and allocates them only when needed.

That cuts waste and lets more requests fit on the same GPU.

𝟯) 𝗣𝗿𝗲𝗳𝗶𝘅 𝗰𝗮𝗰𝗵𝗶𝗻𝗴
A lot of requests share the same prompt start:

• system prompts
• few-shot examples
• RAG context
• conversation history

Prefix caching computes that shared prefix once, then reuses the KV cache.

Less recomputation. More throughput.

𝟰) 𝗦𝗽𝗲𝗰𝘂𝗹𝗮𝘁𝗶𝘃𝗲 𝗱𝗲𝗰𝗼𝗱𝗶𝗻𝗴
Generation is still sequential by default.

Speculative decoding speeds it up by:

• proposing several tokens with a small drafter
• verifying them in one pass with the main model

If the guess is right, you accept multiple tokens at once.

The result:

• lower latency
• higher throughput
• better GPU utilisation