Tag

Inference

Everything tagged Inference, most recent first.

Portrait of Sachin Gupta rendered in binary

Inference

6 entries

Writing· Speculative Decoding
In the Machine Room
The last five parts were the recipe. This one is the kitchen at dinner rush: the two engineering tricks that get DSpark running inside a real serving system, and the speedups it delivers there.
Part six of the speculative decoding series, the finale. Parts one to five were the idea. This one is the engineering: two tricks that get DSpark running inside a live serving system. First, do not ship the whole dictionary of scores between machines while training the guesser, ship the compact summary and do the last step locally. Second, a scheduler that cannot wait, because real GPU speed rises in jagged steps and the pipeline cannot pause, so it predicts the batch size ahead of time. Plus the real production speedups. Plain words, an analogy, a diagram, and two auto-playing visualizers.
- Inference
By Sachin Gupta9 Jul 20267 min read
Writing· Speculative Decoding
The Expediter
DSpark's last machine, on its own: how it decides exactly how much of the draft to check, across everyone it is serving at once, like loading a delivery van to its sweet spot.
Part five of the speculative decoding series, and the last of three that open up DSpark one machine at a time. This one is the expediter, the paper's greedy scheduler. It decides how much of the draft the big model should check, across everyone it is serving at once, and the rule is neat: pool every guessed word, sort the most likely to pass first, and keep loading the batch while the words-per-second keeps climbing, then stop at the peak. Explained with a delivery-van analogy, a diagram, and an auto-playing visualizer. Short, one idea, and it wraps the three machines.
- Inference
By Sachin Gupta8 Jul 20265 min read
Writing· Speculative Decoding
The Honesty Check
DSpark's second machine, on its own: the AI's confidence scores are overconfident, like a forecaster who says 90% but is right 70% of the time, so DSpark corrects them to match reality.
Part four of the speculative decoding series, and the second of three that open up DSpark one machine at a time. This one is the honesty check. DSpark puts a confidence number on every guessed word, but those numbers run high, the same way a weather forecaster who says 90% is often right only 70% of the time. Calibration shaves them down until they match reality, without changing their order, which is what lets the next machine trust them. Plain words, a worked example, a diagram, and an auto-playing visualizer. Short, one idea only.
- Inference
By Sachin Gupta7 Jul 20265 min read
Writing· Speculative Decoding
The Cheat Sheet
DSpark's first fix, on its own: how a tiny cheat sheet, the same trick your phone uses, makes each guessed word fit the one before it.
Part three of the speculative decoding series, and the first of three that open up DSpark one machine at a time. This one is the cheat sheet, the paper's Markov head: the same trick your phone's predictive text uses, where the word you just typed nudges the next one. It explains what a score even is, why the fast guesser is blind, and how a small nudge breaks the tie, with a worked example, a diagram, and an auto-playing visualizer. Short, one idea only.
- Inference
By Sachin Gupta6 Jul 20265 min read
Writing· Speculative Decoding
Draft Better, Verify Smarter
A fast drafter gets sloppy toward the end, and checking its whole guess wastes the big model's time. DSpark makes two small changes that fix both, and the output never changes.
Part two of the speculative decoding series, in plain language. Part one left a dilemma: a draft model is either accurate but slow or fast but sloppy, and checking a long shaky draft wastes the big model's time. This part walks through DSpark, an open method from DeepSeek, one small change at a time. First, let each drafted word see the word before it, so the fast drafter stops contradicting itself. Second, score each word's chance of surviving, multiply those scores into a running number that only falls, and stop checking the moment it stops being worth it. Two interactive visualizers, one of them auto-playing through the whole cycle, and the production numbers, with no change to the output.
- Inference
By Sachin Gupta5 Jul 20268 min read
Writing· Speculative Decoding
Draft, Then Verify
Writing one token at a time makes a large model slow. Speculative decoding speeds it up by letting a small model guess ahead.
Part one of a series on speculative decoding, the trick that lets a large language model generate text faster with no change to the output. This part covers why one-token-at-a-time is slow, the lossless draft-and-verify rule that fixes it, and the real difficulty: the draft model. It walks through the two families of draft models and the flaw in each, autoregressive drafters that are accurate but slow, and parallel drafters that are fast but decay down the block. With two interactive visualizers. Part two covers how DeepSeek's DSpark gets the best of both.
- Inference
By Sachin Gupta4 Jul 20266 min read

Inference

In the Machine Room

The Expediter

The Honesty Check

The Cheat Sheet

Draft Better, Verify Smarter

Draft, Then Verify