All activity

Cache Invalidation: Keeping a Cache Honest

A cache miss is slow but correct. A stale cache hit is fast and wrong, and wrong is worse. Invalidation is the work of making sure the copy in the cache still matches the truth in the database. Here are the ways to do it, each with its own analogy and its own visualizer.

A dedicated look at cache invalidation, the hard half of caching. Why a stale hit is worse than a miss, then the four strategies to keep a cache honest: TTL expiry (bounded staleness), write-through (update on write), invalidate-on-write (delete the key and let the next read refill), and versioned or immutable keys (bump the version, never edit in place). Everyday analogies, a hand-drawn stale-window timeline, and visualizers for the delete-on-write cycle and versioned keys. Why the races make it genuinely hard, and a tee-up to the stampede.

• System Design

You Cannot Improve What You Cannot Measure

Operating an LLM system means making a fuzzy, non-deterministic thing measurable.

LLM systems are hard to operate because outputs are open-ended and non-deterministic. LLM ops and eval is the discipline that makes them measurable: evals as tests, tracing, regression gates, and a flywheel that turns production failures into your next test set.

• LLM Ops & Eval

Cache Eviction: What to Forget When It's Full

A cache is small on purpose, so it is almost always full. To store one more thing it has to throw one thing out, and the whole art is choosing which. Three policies answer that question in three different ways: LRU by recency, LFU by frequency, TTL by age. Here is each one, with its own analogy and its own visualizer.

A dedicated look at cache eviction. Why a cache must forget, then the three policies each with its own everyday analogy and interactive visualizer: LRU (recency, a desk), LFU (frequency, a bestseller shelf), and TTL (age, milk with an expiry date). A worked example for each, a hand-drawn diagram showing the same cache evict three different items, guidance on which to use when, and what Redis and memcached run in production.

• System Design

The Six Primitives of MCP

Tools, resources, prompts, sampling, roots, elicitation, organized by who is in control.

The heart of MCP is six primitives: three the server offers (tools, resources, prompts) and three the client offers (sampling, roots, elicitation). This piece defines each precisely, with method names and the one question that organizes them, plus what the 2025-11-25 revision added around them: async tasks, tool-calling in sampling, richer elicitation, and icons.

• MCP

ANN at Scale: HNSW

Give up on exact, and you can search a billion vectors in milliseconds. This is the graph that makes it happen.

Part five of the search series. Exact kNN is O(N) and hopeless at scale, so production vector search gives up exactness for speed with approximate nearest neighbor. This explains the dominant method, HNSW: navigable small-world graphs, greedy search, the layer hierarchy, and the M / efConstruction / ef knobs, with an interactive graph-search explorer.

• Search

Caching: Don't Do the Same Work Twice

The fastest work is the work you never do. A cache keeps a copy of an answer close by, so the next time someone asks, you hand it over instead of computing it all again. Here is how caches actually work: where they live, how reads and writes flow through them, and why they have to forget things.

A from-the-ground-up look at caching. Why it works (keep the answer close), where caches live (browser, CDN, app, database), the cache-aside read pattern (the hit and the miss), the three write patterns (through, back, around), and eviction (why a full cache must forget, with LRU). Everyday analogies, a hand-drawn diagram, and a visualizer per idea. The hard parts (invalidation, stampedes) and CDNs get their own pieces next.

• System Design

Speaking at DeveloperWeek

Generating working MCP servers straight from the OpenAPI specs you already have: the mechanical field mapping, why it takes a library and not a script, the production traps, and the guardrails to set before an agent touches real users.

• MCP

Vector Search (kNN)

When you stop comparing characters and start comparing coordinates, search can finally match meaning.

Part four of the search series. Keyword and fuzzy search match characters, so they miss synonyms. Vector search embeds text into geometry, where similar meaning becomes nearby points. This covers embeddings, the two distance metrics (cosine vs euclidean), exact k-nearest-neighbor search, and the scaling wall that forces the approximate methods in the next part. With an interactive kNN explorer.

• Search

How a Load Balancer Sticks

Sometimes you want the opposite of spreading traffic evenly: send the same client, or the same request, to the same server every time. That is stickiness, and it runs from a simple hash all the way to the elegant trick of consistent hashing.

A dedicated look at the sticky load-balancing rules, the ones that send the same key to the same server. Why you want it (sessions and caches), then each rule with its own analogy and visualizer: source-IP hash, URL hash, the reshuffling problem, consistent hashing (the ring), Maglev (a fast table), and cookie-based sticky sessions. Part two of two; part one covered the even-spreading rules.

• System Design

How a Load Balancer Spreads the Load

The load balancer's first job is picking which server gets each request. There is a small family of rules for that, from dealing cards in turn to glancing at two servers and choosing the emptier. Here they are, one at a time.

A dedicated look at the load-balancing algorithms that spread traffic across servers. The static-vs-dynamic split, then each rule in turn with its own analogy and visualizer: round-robin, weighted round-robin, least-connections, least-response-time, and power-of-two-choices. Part one of two; part two covers the sticky ones (hashing and affinity).

• System Design