What vLLM Fixes and What It Can't

A question that follows any discussion of batching: doesn't vLLM solve this?

Short answer: no.
It makes the trade-off a lot better.
It can't make it go away.

A quick recap from yesterday.
A 70B model on an H200 streams about 45 to 50 tokens per second to one user, in practice.
Put 100 users in the same batch: around 2,000 to 2,500 tokens per second in total, but 22 to 25 per user.

Every step reads the weights once, plus every user's KV cache.
More users, more to read, slower step.
That's memory bandwidth, and no serving engine changes it.

What vLLM does is get you much closer to what the hardware can do:

• Continuous batching: requests join and leave at every step.
• PagedAttention: the KV cache in small blocks, no wasted memory, more users fit.
• Chunked prefill: long prompts in pieces, so streams don't freeze when a big RAG prompt lands.
• Prefix caching: shared system prompts and tool definitions computed once.
• FP8 KV cache: halves what each user adds to every step.
• Speculative decoding: faster individual streams when batches are small.

What it can't do is choose your operating point.

Two settings matter most:

• maxnumseqs caps how many requests share a step. Higher means more throughput and slower streams.
• maxnumbatched_tokens caps prompt processing per step. Lower gives smoother streams but a slower first token. Higher does the opposite.

The defaults lean toward throughput, with up to 1,024 concurrent sequences.
Probably not what your latency target wants.

What I do: load-test with real prompt lengths, raise concurrency until p95 inter-token latency reaches the target, and cap it there.
That number, not the engine's default, should set your GPU count.

Do you know what maxnumseqs is set to in production?
And who picked it?

#vLLM #LLMOps #Inference