Draft, Then Verify

A large language model writes one token at a time. Each token needs a full pass through the entire network, so a long answer is a long wait, and the GPU spends most of it barely used. Speculative decoding is the trick that lets the model draft ahead and check its own work in a single pass, producing the exact same text, faster.

Imagine a slow, careful head chef and a fast line cook. Instead of the head chef plating every dish herself, the line cook races ahead and plates the next few, and the head chef checks the whole run in one glance: she keeps the plates that are right, fixes the first one that is wrong, and sends them out. The line cook is quick but rough; the head chef has the final say on quality. Between them they send out exactly what the head chef would have plated alone, in a fraction of the time. That is speculative decoding.

Why one token at a time is slow

Language models are autoregressive: each token is a full forward pass over billions of parameters, conditioned on every token before it. So latency grows with the length of the answer. Worse, generating a single token barely uses the GPU, because the work is dominated by reading the model's weights from memory, not by arithmetic. The hardware sits mostly idle. The bottleneck is not compute, it is the sequential dependency: you cannot start token N plus one until token N is finished.

Draft, then verify

Speculative decoding breaks that sequence with two models working together:

• a small, fast draft model proposes a block of several tokens at once,
• the large target model verifies the whole block in a single forward pass.

The target checks each drafted token against its own distribution, accepts the longest correct prefix, and either corrects the first wrong token or, if the whole block was right, adds one free bonus token. Then the cycle repeats from there.

The crucial part is that this is lossless. The acceptance rule (accept each token with probability equal to the target's probability over the draft's, capped at one, and at the first rejection stop and resample a replacement) is built so the final text follows exactly the target model's own distribution. You get the same output you would have gotten the slow way, using fewer expensive target passes to produce it. Speculative decoding buys speed, not a different answer.

The speed comes down to a single number: how many tokens the target accepts per cycle, call it the acceptance length. Per-token latency is roughly the drafting time plus the verification time, divided by that acceptance length. So there are exactly three ways to go faster: draft faster, draft better (accept more per cycle), or verify smarter.

Here is one cycle. The model has written "the best way to learn is to practice." The draft model guesses the next few tokens, the target verifies them all in one pass, keeps the correct prefix, and corrects the first miss. Step through it:

One target pass committed three tokens instead of one. Multiply that across a whole answer, and that is the speedup.

The catch: the draft model is the hard part

If drafting were free and perfect, speculative decoding would be a solved problem. It is not, because the draft model sits on a hard tradeoff, and there are two families, each with a flaw.

Autoregressive drafters (like EAGLE) generate the draft tokens one at a time, each conditioned on the last, just like the target does. Quality is high and they accept a lot per cycle. But the drafting itself is sequential, so its cost grows with the block size. That forces you into short blocks, which caps how much you can win per cycle.

Parallel drafters (like DFlash) produce the whole block in a single pass, so drafting is fast no matter how long the block is. The catch: every position is predicted independently, blind to the others. When two continuations are both plausible, say "of course" and "no problem," a parallel drafter can emit the crossbred "of problem." Because the tokens cannot see each other, quality falls off quickly toward the end of the block. Call it suffix decay.

Here is the parallel failure up close, playing out on its own: two replies both fit, and the parallel drafter crosses them into something that does not.

There is a second, sneakier cost that shows up in production. When you serve many users at once, the target model verifies in batches, and batch capacity is precious. Verifying a long block full of tokens that are probably going to be rejected burns capacity other requests could have used. Under heavy load, drafting long and verifying everything can actually lower total throughput.

So the draft model has to be fast and accurate, and the verification has to be worth its cost even when the server is busy. Autoregressive drafters give up speed; parallel drafters give up quality and waste verification. Getting both at once is the real problem.

Where this goes

Speculative decoding is one of the cleanest wins in LLM serving: a real speedup with no change to the output, as long as the draft model is good enough. The frontier is no longer the acceptance rule, which is settled. It is the drafter, and the verification: how to draft a long block that stays coherent to the end, and how to verify only the parts worth verifying when the server is under load.

Part two takes up exactly that, through a recent open-source method from DeepSeek, DSpark, released with its training code as DeepSpec, which keeps a parallel drafter's speed while cutting its suffix decay, and schedules verification by confidence so the target never wastes its capacity. The next time a chatbot's answer streams out faster than it seems like it should, some version of draft-and-verify is usually why.