Load Balancers, Up Close
A deep look at the one box every large system has: how a load balancer chooses a server, checks their health, the difference between L4 and L7, and who balances the balancer.
A dedicated look at load balancers, the building block introduced in the request-path overview. How it chooses a server (round-robin, least-connections, weighted, sticky), how health checks notice a dead server and route around it, the difference between layer 4 and layer 7 balancing (route by the envelope or open the letter), and how the load balancer itself avoids being a single point of failure. Everyday analogies, a hand-drawn diagram, and a visualizer per idea.
In the last piece we followed a request to a server and met the load balancer, the host at the door (how a request finds a server). It does far more than seat diners, and it comes up in almost every system design, so this whole piece is just about it.
Picture a popular restaurant with one entrance but a big dining room and many waiters. If everyone piled in and picked their own table, some waiters would be swamped and others idle, and if a waiter went home nobody would notice until a table sat forgotten. So there is a host at the door. That host is a load balancer.
In a system, one web address is served by many identical servers, not one. The load balancer sits in front of them behind that single address, and it does four jobs: it chooses which server each request goes to, it watches which servers are alive, it can read a request to send it somewhere specific, and it must not become the weak link itself. We will take them one at a time.
How it chooses a server
Back to the host: when a party arrives, how does it pick a table? By one of a small family of rules. Some just spread traffic evenly (deal to each in turn, or send to the emptiest table); others send the same visitor to the same table every time. There are enough of them, each with its own tradeoff, that they get their own two pieces: how a load balancer spreads the load covers the even-spreading rules, and how a load balancer sticks covers the sticky ones (hashing and affinity). Here is a taste, cycling through a few:
round-robin: deal to each server in turn. Either way, a server that stops answering (server 3) is skipped until it recovers.
How it knows a server is down
A good host glances around the room and notices a waiter who has stopped moving, and stops seating their tables until they are back. A load balancer does the same with health checks: every few seconds it quietly pings each server. A healthy server answers. A server that misses a few checks in a row is pulled out of rotation, so no request is ever sent to a dead machine, and when it starts answering again it is put back. This is why a single server can crash and your users never notice.
Every few seconds the load balancer pings each server. All three answer, so all three take requests.
Reading the envelope, or the letter
Here is where load balancers split into two kinds, and it is simpler than it sounds. Think of a mailroom sorter. One sorter just reads the address on the outside of the envelope and sends it to the right floor, fast, without ever opening it. The other opens the letter, reads what is inside, and routes based on the contents.
A layer 4 load balancer is the first sorter. It looks only at the address and port, not the contents, and forwards the connection. It is very fast and knows nothing about what you asked for. A layer 7 load balancer is the second sorter. It reads the actual request, so it can route by content: send anything starting with /images to the image servers, /checkout to the payment servers, and so on. It does more work per request but it can be much smarter. Most modern setups use layer 7 at the edge for that smartness.
Layer 4 reads only the address, never the contents, and forwards fast to any server. Simple and quick.
Who balances the balancer?
One worry remains. If every request goes through the load balancer, and the load balancer dies, the whole system is down. It would be like a restaurant with a single host who faints: nobody can get seated, no matter how many waiters are idle.
So the load balancer is never truly alone. There are always at least two, a main one and a standby that takes over instantly if the main one fails, and the single address is pointed at whichever is alive (using tricks at the DNS and network layer). The rule of system design is showing up already: anything that every request depends on must not be a single point of failure.
Where this goes
That is the load balancer end to end: it chooses a server by a strategy, health-checks the pool and routes around the dead, reads requests when it needs to (layer 7) or just forwards them fast (layer 4), and keeps a twin so it is never the weak link. Next we make the whole thing faster by not doing the same work twice: caching and CDNs.
