AI
Notes and how-tos on building with LLMs — prompting, agents, retrieval, and evaluation. Written to help other engineers ship AI features that hold up in production.
46 articles
Follow the badge, the folder labels, and the briefing packet.
Six illustrated lessons on identity, document permissions, retrieval, revocation, protected responses, and a runnable local service.
Grading a RAG answer is grading an open-book exam. Two separate questions: did it find the right pages, and did the answer actually use them.
Part eight of the RAG series. A wrong RAG answer has two possible culprits, and one number cannot tell them apart. Evaluation splits like grading an open-book exam: retrieval quality (context precision, did it bring back relevant pages and rank them first; context recall, did it miss any needed page) and generation quality (faithfulness, did the answer only say what the pages support; response relevancy, did it address the question asked). This defines a classic four-metric starting set from Ragas, each with its formula and a worked example, explains which need a ground-truth reference and which use an LLM as judge, and why an LLM-judge score is a moving target. Everyday analogies, a hand-drawn diagram, and a visualizer per metric.
A demo answers one question once. Production answers forever, over a corpus that never stops changing.
Part seven of the RAG series, and the first on running RAG in production. A demo retrieves from a fixed pile of documents and answers one question. Production may retrieve from a corpus that is changing continuously, so the hard part is often not the answer, it is keeping the index true to the source: continuous ingestion and incremental re-indexing for freshness, deletion that removes a document's chunks so the model cannot cite what no longer exists, a migration plan for the day you change the embedding model and old vectors no longer share a coordinate system with new ones, and a trace on every answer so you can see why it said what it said. Grounded in the RAGOps paper. Everyday analogies, a hand-drawn diagram, and interactive visualizers.
The last five parts were the recipe. This one is the kitchen at dinner rush: the two engineering tricks that get DSpark running inside a real serving system, and the speedups it delivers there.
Part six of the speculative decoding series, the finale. Parts one to five were the idea. This one is the engineering: two tricks that get DSpark running inside a live serving system. First, do not ship the whole dictionary of scores between machines while training the guesser, ship the compact summary and do the last step locally. Second, a scheduler that cannot wait, because real GPU speed rises in jagged steps and the pipeline cannot pause, so it predicts the batch size ahead of time. Plus the real production speedups. Plain words, an analogy, a diagram, and two auto-playing visualizers.
DSpark's last machine, on its own: how it decides exactly how much of the draft to check, across everyone it is serving at once, like loading a delivery van to its sweet spot.
Part five of the speculative decoding series, and the last of three that open up DSpark one machine at a time. This one is the expediter, the paper's greedy scheduler. It decides how much of the draft the big model should check, across everyone it is serving at once, and the rule is neat: pool every guessed word, sort the most likely to pass first, and keep loading the batch while the words-per-second keeps climbing, then stop at the peak. Explained with a delivery-van analogy, a diagram, and an auto-playing visualizer. Short, one idea, and it wraps the three machines.
DSpark's second machine, on its own: the AI's confidence scores are overconfident, like a forecaster who says 90% but is right 70% of the time, so DSpark corrects them to match reality.
Part four of the speculative decoding series, and the second of three that open up DSpark one machine at a time. This one is the honesty check. DSpark puts a confidence number on every guessed word, but those numbers run high, the same way a weather forecaster who says 90% is often right only 70% of the time. Calibration shaves them down until they match reality, without changing their order, which is what lets the next machine trust them. Plain words, a worked example, a diagram, and an auto-playing visualizer. Short, one idea only.
DSpark's first fix, on its own: how a tiny cheat sheet, the same trick your phone uses, makes each guessed word fit the one before it.
Part three of the speculative decoding series, and the first of three that open up DSpark one machine at a time. This one is the cheat sheet, the paper's Markov head: the same trick your phone's predictive text uses, where the word you just typed nudges the next one. It explains what a score even is, why the fast guesser is blind, and how a small nudge breaks the tie, with a worked example, a diagram, and an auto-playing visualizer. Short, one idea only.
A fast drafter gets sloppy toward the end, and checking its whole guess wastes the big model's time. DSpark makes two small changes that fix both, and the output never changes.
Part two of the speculative decoding series, in plain language. Part one left a dilemma: a draft model is either accurate but slow or fast but sloppy, and checking a long shaky draft wastes the big model's time. This part walks through DSpark, an open method from DeepSeek, one small change at a time. First, let each drafted word see the word before it, so the fast drafter stops contradicting itself. Second, score each word's chance of surviving, multiply those scores into a running number that only falls, and stop checking the moment it stops being worth it. Two interactive visualizers, one of them auto-playing through the whole cycle, and the production numbers, with no change to the output.
When a document has a structure, stop shredding it. Navigate it.
Part six of the RAG series. The default RAG recipe chops a document into chunks, embeds them, and retrieves by vector similarity. For a long, structured document that throws away the one thing that made it readable: its structure. Vectorless RAG keeps the document whole, builds a table-of-contents tree, and has a model reason its way to the right section, the way a human expert uses a table of contents. This covers why similarity is not relevance, how tree-navigation retrieval works, where it fails, when to reach for it, and when plain vector search is still the right tool. With an interactive navigator.
Writing one token at a time makes a large model slow. Speculative decoding speeds it up by letting a small model guess ahead.
Part one of a series on speculative decoding, the trick that lets a large language model generate text faster with no change to the output. This part covers why one-token-at-a-time is slow, the lossless draft-and-verify rule that fixes it, and the real difficulty: the draft model. It walks through the two families of draft models and the flaw in each, autoregressive drafters that are accurate but slow, and parallel drafters that are fast but decay down the block. With two interactive visualizers. Part two covers how DeepSeek's DSpark gets the best of both.
