Two vLLM Settings Your Users Can Feel

Last week I wrote about what vLLM fixes and what it leaves to you.
Two settings came up.
They deserve their own post.

They shape what users actually feel.
And they're often left on their defaults.

• maxnumseqs caps how many requests share the GPU at each step.
• maxnumbatched_tokens caps how many tokens the GPU processes in that step.

If a step were a bus, the first is the number of passengers.
The second is the weight limit.

You need both because requests weigh very different amounts.
A request streaming its answer adds one token per step.
A request that just arrived needs its whole prompt processed: thousands of tokens.

An example with a budget of 8,192 tokens and 100 users streaming:

• the streams use 100 tokens
• roughly 8,000 are left for new prompts
• a 20,000-token RAG prompt gets processed in three slices

Each setting controls a different trade-off.

maxnumseqs is throughput against speed per user.
Raise it and you serve more people per GPU, each a bit slower.

maxnumbatched_tokens is first-token speed against smooth streaming.

• Lower (around 2,000): streams stay smooth, long new prompts wait longer for their first token.
• Higher (around 16,000): first tokens come faster, but every big prompt makes everyone else's stream stutter.

For RAG and agents with long prompts, the second matters as much as the first.

Recent defaults: 1,024 sequences, 8,192 tokens for online serving.
A throughput-friendly starting point.
Not a latency target.

How I tune them:

• FP8 KV cache and prefix caching first. They're free.
• Load-test with your real prompt lengths.
• Lower maxnumseqs until p95 inter-token latency meets the target.
• Adjust maxnumbatched_tokens depending on whether users notice stutter or a slow first token.
• Write the values in the runbook, with the test that justified them.

Two command-line flags.
Minutes to change.
A good reason to make sure someone actually chose them.

Has your team tuned either of these, or are you running on defaults?

#vLLM #LLMOps #Inference