How a Load Balancer Spreads the Load
Part one of load balancing strategies: the algorithms that spread requests evenly, round-robin, weighted, least-connections, least-response-time, and the power of two choices, each with its own visualizer.
A dedicated look at the load-balancing algorithms that spread traffic across servers. The static-vs-dynamic split, then each rule in turn with its own analogy and visualizer: round-robin, weighted round-robin, least-connections, least-response-time, and power-of-two-choices. Part one of two; part two covers the sticky ones (hashing and affinity).
The load balancer piece introduced its four jobs (load balancers, up close). The very first, choosing which server gets each request, is a small topic on its own, so here it is. This part covers the rules that spread traffic evenly; the next covers the sticky ones.
Back to the restaurant host at the door. A party arrives; which table do they get? There is no single answer, just a handful of simple rules, and each makes a different tradeoff. Before the rules, one split that sorts them all.
Static rules pick a server by a fixed formula (whose turn is it, what is its weight) without looking at how busy anyone is. They are cheap and predictable. Dynamic rules watch live signals (how many connections, how fast each server is answering) and steer toward the least loaded. They are smarter but need bookkeeping. Keep that split in mind as we go.
Round-robin
Picture dealing cards around a table: one to each player in turn, then back to the first. That is round-robin. The load balancer keeps a pointer and sends each new request to the next server in order, around and around. It is the simplest rule there is, and it spreads requests perfectly evenly when the servers are identical and every request costs about the same.
Round-robin: each server gets the next request in turn, like dealing cards. Even and simple, but it ignores how busy each one is.
Its weakness is what it ignores: it does not care that one server is bogged down with a slow request. A busy server still gets its turn.
Weighted round-robin
Now imagine some players have a bigger table and can take more cards. Weighted round-robin deals in proportion to a weight you give each server, so a machine with twice the capacity gets twice the share. It is how you mix server sizes, and how you do canary releases: give the new version a weight of 1 and the old one a weight of 9, and only a tenth of traffic tries the new one.
Weighted round-robin: bigger servers get a bigger share of the rotation. Server 3 has weight 3, so it is dealt three times as often as server 1.
The catch is that the weights are set by hand and do not react to real load. Wrong weights just entrench the imbalance.
Least connections
Here is the first dynamic rule. A good host seats the next party at the emptiest table, not just the next one in line. Least-connections sends each request to the server currently handling the fewest connections. It shines when requests are long-lived and uneven, like database sessions or WebSockets, where a plain rotation would pile long jobs onto an already-busy server.
Least connections: send the request to whichever server is handling the fewest right now. Good when connections are long-lived and uneven.
Its blind spot: connection count is a rough guess at real work. A server with few but very heavy connections still looks idle.
Least response time
Sharper still: send the request to the server that is answering fastest right now. Least-response-time watches how quickly each server replies and favors the quick ones, which naturally steers traffic away from a server that is struggling, even before it fails a health check. It is the most responsive to trouble, but it needs constant latency measurement, and noisy numbers can make it flip-flop.
Least response time: send to the server answering fastest right now. It naturally steers away from a slow or struggling server.
Power of two choices
The clever one, and the modern default. Checking every server to find the truly least-busy is expensive when there are thousands, and when many load balancers do it at once they can all pile onto the same "least-busy" server. The trick: do not check them all. Pick just two servers at random, and send the request to the less busy of the two. That tiny bit of choice gets you almost the evenness of checking everyone, for almost none of the cost, and it avoids the herd.
Power of two choices: pick two servers at random, then send to the less busy of the two. Almost as good as checking them all, at a fraction of the cost.
You will see this under different names: HAProxy calls its random this, Envoy uses it for its
least-request policy, and NGINX writes it random two least_conn.
Where this goes
Those are the rules that spread traffic evenly: round-robin and its weighted form (static), and least-connections, least-response-time, and power-of-two-choices (dynamic). The other half of load balancing does the opposite: it tries to send the same client, or the same request, to the same server every time. That is stickiness, and it is part two: how a load balancer sticks.
