Speculative Decoding

Speculative decoding speeds up LLM generation by exploiting a key asymmetry: 𝘁𝗼𝗸𝗲𝗻 𝗴𝗲𝗻𝗲𝗿𝗮𝘁𝗶𝗼𝗻 𝗶𝘀 𝘀𝗲𝗾𝘂𝗲𝗻𝘁𝗶𝗮𝗹, 𝗯𝘂𝘁 𝘁𝗼𝗸𝗲𝗻 𝘃𝗲𝗿𝗶𝗳𝗶𝗰𝗮𝘁𝗶𝗼𝗻 𝗶𝘀 𝗽𝗮𝗿𝗮𝗹𝗹𝗲𝗹.

• 𝗠𝗮𝗶𝗻 𝗯𝗼𝘁𝘁𝗹𝗲𝗻𝗲𝗰𝗸: Standard autoregressive decoding is memory-bandwidth bound. Each new token requires another full pass over the model weights in HBM, so GPUs spend much of their time waiting on memory rather than doing math.
• 𝗖𝗼𝗿𝗲 𝗶𝗱𝗲𝗮: Use a cheap draft proposer to guess several future tokens, then have the large target model verify them all at once in a single parallel forward pass.
• 𝗛𝗼𝘄 𝗶𝘁 𝘄𝗼𝗿𝗸𝘀: The draft model proposes K tokens; the target model checks them left-to-right, accepts matching tokens, and replaces any mismatch with a token sampled from the target distribution.
• 𝗪𝗵𝘆 𝗶𝘁’𝘀 𝗹𝗼𝘀𝘀𝗹𝗲𝘀𝘀: With modified rejection sampling, the final output distribution is identical to sampling from the target model alone, so quality is preserved exactly.

• 𝗗𝗿𝗮𝗳𝘁 𝗺𝗲𝗰𝗵𝗮𝗻𝗶𝘀𝗺𝘀:

1. 𝗦𝗺𝗮𝗹𝗹 𝗱𝗿𝗮𝗳𝘁 𝗺𝗼𝗱𝗲𝗹 + 𝗹𝗮𝗿𝗴𝗲 𝘁𝗮𝗿𝗴𝗲𝘁 𝗺𝗼𝗱𝗲𝗹 — good accuracy, but extra VRAM.
2. 𝗣𝗿𝗼𝗺𝗽𝘁 𝗹𝗼𝗼𝗸𝘂𝗽 / 𝗻-𝗴𝗿𝗮𝗺 𝗺𝗮𝘁𝗰𝗵𝗶𝗻𝗴 — no extra model, very fast, best when output overlaps with context (𝗲.𝗴., 𝗥𝗔𝗚).
3. 𝗦𝗲𝗹𝗳-𝘀𝗽𝗲𝗰𝘂𝗹𝗮𝘁𝗶𝗼𝗻 𝗵𝗲𝗮𝗱𝘀 — extra heads on the target model predict future tokens without a separate draft model.

• 𝗣𝗿𝗼𝗱𝘂𝗰𝘁𝗶𝗼𝗻 𝘁𝗿𝗮𝗱𝗲-𝗼𝗳𝗳:
• At 𝗹𝗼𝘄 𝗤𝗣𝗦, speculative decoding usually helps a lot because GPUs have idle compute.
• At 𝗵𝗶𝗴𝗵 𝗤𝗣𝗦, it can hurt latency because the system is already compute-saturated, so extra verification adds overhead.
• 𝗣𝗿𝗮𝗰𝘁𝗶𝗰𝗮𝗹 𝘁𝗿𝗲𝗻𝗱: Modern systems use 𝗱𝘆𝗻𝗮𝗺𝗶𝗰 𝘀𝗽𝗲𝗰𝘂𝗹𝗮𝘁𝗶𝗼𝗻, adjusting speculative length based on current load.

Speculative decoding trades a little extra draft work for fewer expensive target-model passes, giving substantial latency gains when the GPU is memory-bound and underutilized.