Service Level Objectives

Why your LLM app looked great in the notebook, but failed the moment real users hit it?

The average latency was fine.

The p99 was not.

That is when you should stop treating “fast enough” as a feeling and start defining SLOs properly.

For production LLMs, you are always balancing three things: 𝗮𝗰𝗰𝘂𝗿𝗮𝗰𝘆, 𝗹𝗮𝘁𝗲𝗻𝗰𝘆, 𝗮𝗻𝗱 𝗰𝗼𝘀𝘁. You can usually optimize two. The third will move against you.

What I do in practice:

• Define the user experience first: chatbot, RAG, agent, support flow — each has different tolerance for delay.
• Measure the right metrics: 𝗧𝗧𝗙𝗧, 𝗜𝗧𝗟, and 𝗲𝗻𝗱-𝘁𝗼-𝗲𝗻𝗱 𝗹𝗮𝘁𝗲𝗻𝗰𝘆. A single average hides too much.
• Set targets before benchmarking: for an e-commerce chatbot, I would expect something like 𝗧𝗧𝗙𝗧 < 𝟮𝟬𝟬𝗺𝘀 and 𝗜𝗧𝗟 < 𝟱𝟬𝗺𝘀 at 𝗽𝟵𝟵. For RAG, I can usually relax that to around 𝗧𝗧𝗙𝗧 < 𝟯𝟬𝟬𝗺𝘀, 𝗜𝗧𝗟 < 𝟭𝟬𝟬𝗺𝘀, and < 𝟯𝘀 end-to-end.
• Watch the tails, not just the mean. If p95 or p99 spikes, users feel it immediately — even when dashboards look healthy.
• Use LLM-specific tooling like 𝗚𝘂𝗶𝗱𝗲𝗟𝗟𝗠 to validate streaming behavior under realistic traffic, not just synthetic request timing.

Prototype success means very little if you have not defined what “good” looks like in production.

How do you set SLOs for your LLM systems today?