Disaggregated LLM Serving

Disaggregated LLM serving looks complicated.

But the core idea is simple:
Don’t make one GPU do two very different jobs.

Disaggregated LLM serving splits LLM inference into two separate GPU pools:

• 𝗣𝗿𝗲𝗳𝗶𝗹𝗹 (𝗣-𝗽𝗼𝗼𝗹): Handles the prompt/context phase. It is 𝗰𝗼𝗺𝗽𝘂𝘁𝗲-𝗯𝗼𝘂𝗻𝗱, so it benefits from highly parallel execution.
• 𝗗𝗲𝗰𝗼𝗱𝗲 (𝗗-𝗽𝗼𝗼𝗹): Handles token-by-token generation. It is 𝗺𝗲𝗺𝗼𝗿𝘆-𝗯𝗮𝗻𝗱𝘄𝗶𝗱𝘁𝗵-𝗯𝗼𝘂𝗻𝗱, so it benefits from specialized memory-efficient kernels and high tensor parallelism.

𝗪𝗵𝘆 𝘁𝗵𝗶𝘀 𝗵𝗲𝗹𝗽𝘀

• Reduces interference between prompt processing and generation
• Lowers 𝗧𝗶𝗺𝗲-𝗧𝗼-𝗙𝗶𝗿𝘀𝘁-𝗧𝗼𝗸𝗲𝗻 (𝗧𝗧𝗙𝗧)
• Improves overall GPU utilization and throughput
• Avoids duplicated work by reusing cached prompt prefixes

𝗠𝗮𝗶𝗻 𝗰𝗼𝗺𝗽𝗼𝗻𝗲𝗻𝘁𝘀

• 𝗢𝗿𝗰𝗵𝗲𝘀𝘁𝗿𝗮𝘁𝗼𝗿 / 𝗦𝗺𝗮𝗿𝘁 𝗥𝗼𝘂𝘁𝗲𝗿: Receives requests and routes them to the best prefill worker, often using 𝗞𝗩-𝗰𝗮𝗰𝗵𝗲-𝗮𝘄𝗮𝗿𝗲 𝗿𝗼𝘂𝘁𝗶𝗻𝗴
• 𝗣𝗿𝗲𝗳𝗶𝗹𝗹 𝘄𝗼𝗿𝗸𝗲𝗿𝘀: Process the prompt, generate the first token, and build the KV cache
• 𝗗𝗲𝗰𝗼𝗱𝗲 𝘄𝗼𝗿𝗸𝗲𝗿𝘀: Continue generation from the transferred KV cache
• 𝗞𝗩 𝗰𝗮𝗰𝗵𝗲 𝘁𝗿𝗮𝗻𝘀𝗽𝗼𝗿𝘁 𝗮𝗴𝗲𝗻𝘁: Moves cache blocks efficiently between P and D workers using RDMA/NIXL-like mechanisms

𝗥𝗲𝗾𝘂𝗲𝘀𝘁 𝗳𝗹𝗼𝘄

1. Router receives the prompt and selects suitable P/D workers
2. Prefill worker processes the input and creates KV cache
3. Cache blocks are transferred to the decode worker
4. Decode worker continues generation and streams tokens until EOS

𝗖𝗼𝗺𝗺𝗼𝗻 𝗳𝗿𝗮𝗺𝗲𝘄𝗼𝗿𝗸𝘀

• 𝗹𝗹𝗺-𝗱: Kubernetes-native disaggregated serving framework built on vLLM, using custom scheduling and NIXL-based cache transport
• 𝗗𝗶𝘀𝘁𝗦𝗲𝗿𝘃𝗲: Research system that separates prefill and decode SLOs, improving request throughput significantly
• 𝗧𝗲𝗻𝘀𝗼𝗿𝗥𝗧-𝗟𝗟𝗠 / 𝗗𝘆𝗻𝗮𝗺𝗼: NVIDIA’s approach for large-scale disaggregated serving with custom metadata and high-speed interconnects