The Expediter
DSpark's last machine, on its own: how it decides exactly how much of the draft to check, across everyone it is serving at once, like loading a delivery van to its sweet spot.
Part five of the speculative decoding series, and the last of three that open up DSpark one machine at a time. This one is the expediter, the paper's greedy scheduler. It decides how much of the draft the big model should check, across everyone it is serving at once, and the rule is neat: pool every guessed word, sort the most likely to pass first, and keep loading the batch while the words-per-second keeps climbing, then stop at the peak. Explained with a delivery-van analogy, a diagram, and an auto-playing visualizer. Short, one idea, and it wraps the three machines.
The draft is good and its confidence numbers are honest. Now the big model has to check them, and one question is left: how much? Check too little and you miss easy speed. Check too much and you waste the big model on words that were going to be thrown out, and slow everyone else down. DSpark answers this with its last machine, the expediter, the one that decides what actually gets checked.
A quick recap. A big slow AI is sped up by letting a small, fast model guess the next few words while the big one checks them. Part four gave every guessed word an honest number: how likely it is to pass the check. This machine puts those numbers to work.
Think of loading a delivery van. Each parcel you add is one more delivery you might make, but a fuller van is heavier and slower. So more parcels is not always more deliveries per hour: past a point, the extra weight slows you down more than the extra parcel helps. There is a sweet spot. And you load the sure-thing parcels first and the shaky ones last, so the shaky ones are the ones you leave behind when the van is full.
The setup
The big model does not check one draft at a time. It checks in batches, and it is serving many people at once. So at any moment there is a big pile of guessed words, from lots of different requests, each one carrying its honest chance of passing from part four.
The rule
Here is the whole rule. Pool every guessed word from every request, and line them up, the most likely to pass first. Then load them into the batch one at a time. Each word you add is one more word that will probably get through, which is good, but it also makes the batch bigger, and a bigger batch runs slower, which is bad. So DSpark keeps a running score: how many words get through per second. It loads while that score climbs, and stops the moment it starts to fall.
Here is the same idea as a short story, from what was happening to the fix and why it works. The words come from the requests, and the curve is the score, climbing to a peak and falling as the batch gets too full:
The big model checks the guessed words in batches, serving many requests at once. Each word carries an honest chance of passing, from the honesty check.
Why this is safe, and free
Stopping at the peak makes sense: the words near the bottom of the sorted pile are the least likely to pass, so once loading stops paying off, going further mostly wastes the big model's time. On a smoothly behaving machine it never pays to keep going. Real hardware is bumpier than that, which is part of what makes actually shipping this a job of its own.
Two nice things fall out of this. First, the answer never changes: leaving a word out costs a little speed, never correctness, because whatever does get checked is checked against the big model exactly as before. Second, it handles load for free: when the servers are busy, the batch is already big and slow, so the sweet spot comes sooner. Busy means check less; quiet means check deep. Nobody sets a limit by hand.
Where this goes
That is all three machines. The cheat sheet makes the draft fit together, the honesty check makes its confidence trustworthy, and the expediter spends the big model's time exactly where it pays. Together they are DSpark: a fast draft that stays coherent, and a check that never wastes the big model, with the output identical to the slow way. The code and the real math are open in DeepSpec, alongside the paper.
There is one more layer, if you want it. Getting all of this to actually run inside a live serving system is its own engineering problem, full of the low-level tricks that keep a GPU busy. That is the machine room, in part six.
