50 Tokens/s or 2,500: Batching, and Who Pays for Latency

Here's a number that surprised me the first time I worked it out.

A 70B model on an H200 produces about 68 tokens per second for one user on paper.
45 to 50 in practice.
And during that time, the GPU's compute is idle more than 99% of the time.

The reason is memory.
To produce one token, the GPU reads all 70 GB of weights (FP8) from memory.
At 4.8 TB/s, that takes about 14.6 milliseconds.
The arithmetic itself takes 0.07 ms.
The rest is waiting.

Batching is how you get that time back.
If 100 requests share the same forward pass, the weights are read once and used 100 times.

In theory, that goes up to around 13,000 tokens per second.
In practice, each user's KV cache also has to be read at every step, and that part isn't shared.

• 2,000-token contexts: about 100 users, near 3,600 tokens/s on paper, 2,000 to 2,500 measured
• 8,000-token contexts: more like 25 users

Same chip.
Roughly 50 tokens per second, or 2,500.
A 50x difference in cost per token.

The catch: bigger batches make each stream slower.
At batch 100, each user gets 22 to 25 tokens per second instead of 45 to 50.

So the latency target decides the batch size.
The batch size decides the cost.
A latency SLO is a pricing decision.

I try to get teams to sort use cases into tiers:

• Real-time: someone watches text appear and needs a first token within a second.
• Async: "answer within 30 seconds" is fine.
• Batch: overnight, the GPU packed as tightly as it goes.

The price gap between the first and last tier is large.

A real conversation does need a fast first token.
What it rarely needs is 50 tokens per second.
People read around 5 to 7.

Streaming at 15 or 20 is already faster than anyone reads, and it lets far more users share each GPU.
And chat features that aren't conversations, like "summarize this document" or "draft a report", usually belong in the second tier.

What streaming speed does your chat target?
Has anyone compared it to how fast people read?

#Inference #LLMOps #FinOps