Lessons (6, study in order)

Lesson 1 · Why load balancing exists

Beginner · about 8 min

Goal: Understand exactly what problem load balancing solves, and what a VIP is.

A single server can't keep up, and it's scary if it goes down

As traffic grows, a single server runs into two problems:

(1) Can't keep up -- CPU/bandwidth/connection count maxes out
(2) Single point of failure -- if this one server goes down, the service is gone

A load balancer (LB) solves exactly these two things: it spreads the traffic behind one entry-point address (a VIP, Virtual IP) across a group of backend servers, while the client has no idea how many backends there are or which ones they are.

★ VIP: the only address the client knows

The client always connects to just one virtual IP (VIP) -- whether there are 3 backends behind it or 30, adding machines, removing machines, swapping machines is completely invisible to the client.

★ This is a bit like a CNAME in the DNS module (one external name, the target behind it can change) -- but an LB decides in real time, per flow, while DNS decides once, at query time.

Try it: watch traffic get spread out

Go to the traffic simulation page, send a few flows, and watch how they land on different backends.

Key takeaways

• ★ Load balancing solves two things: can't keep up + single point of failure
• ★ The VIP is the only address the client knows -- backends can be added or removed at any time without affecting clients
• Different from DNS's CNAME: an LB decides in real time per flow, not once at query time

Lesson 2 · ★★★ L3/L4/L7: the core difference is decision granularity

Advanced · about 12 min

Goal: This is the core of the whole module: L3 looks at IP, L4 looks at connections, L7 looks at requests.

★★★ It's not just about which header it looks at

It's easy to think the difference between L3/L4/L7 is just which header gets looked at -- that's true, but the more critical point is the decision granularity:

| Layer | Looks at | Decision granularity |
|---|---|---|
| L3 (NLB) | Only IP | Per client -- no matter how many ports are opened, always the same backend |
| L4 (CLB) | 4-tuple (IP+port) | Per connection -- a different connection may land on a different backend |
| L7 (ALB) | HTTP content | Per request -- two requests on the same connection can go to different backends |

★★★ L7 can achieve request-level granularity because it isn't forwarding at all -- it's a reverse proxy: it terminates the client's connection first, parses out the request, and opens a separate connection of its own to ask the backend. Client-to-LB and LB-to-backend are two independent connections.

★★ Why L3/L4 can't reach request-level granularity

L3/L4 are forwarding, not proxying -- a packet arrives and gets sent to some backend according to rules; the LB never terminates the connection, and never parses what request is inside.

From the moment a TCP connection opens to when it closes, every packet in between must go to the same backend -- otherwise the backend receives half of the data out of context, and it simply breaks. So the floor on L3/L4's decision granularity is one connection, and that can't be changed.

Try it: see the granularity difference with your own eyes

Go to the traffic simulation page:
(1) Scenario ① (L3): the same client, two ports -- see whether it's always the same backend
(2) Scenario ③ (L7): the same connection, first send /api/users then /static/logo.png -- see whether they can land on different backends

Key takeaways

• ★★★ Decision granularity: L3=client, L4=connection, L7=request
• ★★ L7 reaches request-level granularity because it's a reverse proxy -- two independent connections
• L3/L4 are forwarding, never terminate the connection, and their granularity floor is one connection

Lesson 3 · ★★ Load balancing algorithms

Advanced · about 10 min

Goal: The most common algorithms, and what scenario each one fits.

The two simplest: round robin / weighted round robin

Round robin: one at a time, A, B, C, A, B, C...

Weighted round robin: used when machines have different capacity -- a machine with weight 3 shows up 3 times more often in the rotation.

★ A limitation both share: neither looks at the backend's actual load -- even if a backend is already buckling under slow queries, round robin still sends it traffic when its turn comes up.

★ Least connections: looks at actual load

Least connections: the new connection goes to whichever backend currently has the fewest connections attached.

★ Closer to real load, but it requires the LB to continuously maintain a connection count for every backend -- a bit more state to keep than round robin.

★★ Hash-based: the same key always gets the same backend

Modulo hashing / consistent hashing: take a key (an IP, or a 4-tuple), compute a hash, and map it to a backend -- the same key always produces the same result.

★ This is where the determinism comes from, unlike round robin / least connections -- the results of round robin and least connections depend on history (how many requests came before), while hash-based algorithms' results depend only on the key itself.

★★ The difference between the two hashing schemes is covered in detail in lesson 5 -- modulo hashing reshuffles heavily when scaling out, consistent hashing does not.

Try it: switch algorithms and compare

Go to the configure load balancer page to switch algorithms, then go back to the traffic simulation page and run the same batch of flows again, comparing the distribution.

Key takeaways

• Round robin/weighted round robin: simple, but doesn't look at actual load
• Least connections: looks at actual load, but has to maintain connection counts
• ★★ Hash-based: deterministic -- the same key always gets the same result, independent of history

Lesson 4 · ★★ Health checks

Advanced · about 8 min

Goal: How the LB knows whether it should keep sending traffic to a given backend.

★★ A health check is just probe traffic the LB sends itself

The LB periodically probes every backend on its own (e.g. sending an HTTP GET /health, or just a plain TCP connection attempt), and uses the probe result to decide whether that backend can currently take traffic.

★★★ Note: this is an ordinary request the LB sends itself, not some signaling protocol syncing state across multiple devices -- it's the same fact as "load balancing has no control plane": an LB only ever has traffic, and a health check is just another kind of traffic, nothing more.

★ Unhealthy backends get automatically removed

Once a backend is judged unhealthy, no new traffic is sent to it (existing old connections on it generally aren't forcibly cut, but the teaching focus here is new traffic).

Once the health check recovers, that backend gets automatically added back -- no manual intervention needed.

Try it: watch traffic route around an unhealthy backend

Current scenario: ④ Health checks: Backend2 has already been marked unhealthy. Go to the traffic simulation page and send several flows in a row, and see whether they all avoid Backend2.

Key takeaways

• ★★★ A health check is probe traffic the LB sends itself, not a control-plane protocol
• ★ Unhealthy = automatically removed, recovered = automatically added back -- no manual intervention
• This is a concrete example of load balancing having no control plane, only traffic

Lesson 5 · ★★ Consistent hashing: why scaling out causes less churn

Advanced · about 10 min

Goal: Modulo hashing reshuffles heavily when scaling out, consistent hashing doesn't -- how does it work.

★★ The problem with modulo hashing: change the denominator, reshuffle everything

Modulo hashing: backend = hash(key) % backend_count.

The problem is exactly that % backend_count -- when the backend count goes from 3 to 4, almost every key's remainder changes (the division's denominator changed), so almost every client ends up switching to a different backend.

★ What does switching backends mean? If these connections are stateful (e.g. a login session cached locally), almost every user's session invalidates the instant you scale out.

★★★ Consistent hashing: throw both backends and keys onto a ring

Draw a large numeric space as a ring (say, 0-10000). Each backend occupies a few points on the ring (hash its name), and each key also has a position on the ring (hash the key itself).

The rule: walk clockwise from the key's position, and the first backend point you hit is the one it maps to.

★★★ Adding a new backend only adds a few new points on the ring -- only the small slice of keys that happened to sit clockwise right before those new points changes its mind, and every other key's assignment is completely unaffected.

Try it: count the churn difference with your own eyes

Current scenario: ⑤ Consistent hashing: first run modulo hashing against 5 clients and remember the mapping; click "add a backend"; run it again and count how many changed.

Switch to consistent hashing, repeat, and compare the churn count.

Key takeaways

• ★★ Modulo hashing: % backend_count -- change the denominator, reshuffle almost everything
• ★★★ Consistent hashing: both backends and keys sit on one ring -- adding a backend only affects a small slice
• Less churn means fewer affected users/sessions when scaling out

Lesson 6 · ★★ Session persistence: why only L7 can do it

Advanced · about 10 min

Goal: How cookie stickiness works, and what it costs.

★★ Some applications need the same user to always land on the same backend

If a backend server stores user state (e.g. a shopping cart, login state) in its own local memory instead of a shared database/cache, that user must return to the same backend every time -- switch to a different server and the state in local memory is gone.

This kind of stickiness is called session affinity.

★★★ Why only L7 can do this

The most common implementation of session persistence stuffs a marker into a cookie -- and reading a cookie requires parsing the HTTP header, an L7-only capability.

The flow:
(1) The first request carries no such cookie -> pick a backend using the normal algorithm, and set a cookie in the response recording which one was picked
(2) The next request comes back carrying that cookie -> regardless of what the algorithm would otherwise pick, it sticks right back to the recorded backend

★ The stickiness L4 can manage is sticking by source IP (fundamentally just IP hashing), which becomes inaccurate for a large group of users sharing the same public IP behind NAT -- this is also why L7 cookie stickiness is more precise: it sticks to the user, not the IP.

Try it: watch stickiness override round robin

Current scenario: ⑥ Session persistence: send a first request (no cookie) and note the cookie you get back; send a second request carrying that cookie -- even if round robin should have moved on to a different backend, it should stick back to the first one.

Key takeaways

• The problem session persistence solves: backends store user state locally
• ★★★ L7 relies on reading cookies for stickiness; L4 can at best stick by source IP (imprecise)
• ★ A more thorough fix is for backends to stop storing state locally -- put it in a shared cache and stickiness becomes unnecessary