Tokens Don't Size GPUs. Memory Does.

263 million tokens a month.
That's about 100 tokens per second.
Less than 2% of one H200.

The production answer was still 4 GPUs. Not 1. Not 0.02.

Here's why the usual formula fails:

GPUs = tokens per month ÷ tokens per second per GPU

It treats an LLM platform like a batch job. It isn't one.

What actually set the GPU count, in order:

1. Model residency

3-4 models must sit hot in VRAM (a 7B for intent and agents, a 30B for RAG, plus embeddings and a reranker). You pay for memory before you serve a single token.

1. Latency SLOs

Interactive chat means first token under 1 second and 30-50 tokens/s per user. That caps batch size, which caps how hard you can push each GPU.

1. High availability

N+1 replicas across 2 availability zones. That alone roughly doubles the count.

1. KV cache

Long RAG and agent contexts (4-6K tokens) eat memory per concurrent user. That, not volume, is what eventually forces more GPUs.

Throughput came last. It always does at enterprise volumes.

The result:
→ 4× H200 for a 30B RAG model
→ 6-8× H200 for a 120B one
→ plus embedding and reranker capacity nobody budgeted for

My rule: size for memory, latency, and failure first. Check throughput last, as a sanity check.

If your capacity plan starts with "tokens per month", you're not sizing a platform. You're sizing a benchmark.

What's the first number you write down when someone asks "how many GPUs do we need"?

#ProductionAI #LLMOps #GPU #AIArchitecture #Inference