Lessons (6, study in order)

Lesson 1 · What's wrong with the traditional tree architecture

Beginner · about 10 min

Goal: First understand what problem spine-leaf actually solves — otherwise its two iron rules look arbitrary.

The traditional 3-tier: core / aggregation / access

This is the architecture data centers used for over a decade: servers connect to access, access uplinks to aggregation, aggregation uplinks to core.

It was designed for north-south traffic (users from outside the network accessing servers):
traffic comes in from the outside, walks down layer by layer to the server, then walks back up.
That was fine back then — servers barely talked to each other.

★ Problem one: east-west traffic overwhelmed it

★ The change happened in application architecture, not in the network.

Virtualization, microservices, distributed storage, big data, AI training — what all of these have in common is that servers need to talk to each other constantly (east-west traffic).

And the way a tree architecture handles east-west traffic is "go all the way up, then all the way back down":
· Same access switch → 1 hop
· Different access, same aggregation → 3 hops
· Different aggregation, through the core → 5 hops

★★ The fatal part isn't "5 hops is slower than 1" — it's that latency depends on which rack the server happens to sit in.
In an era where VMs can migrate at any time, that's unacceptable: two instances of the same app can have low latency today and, after being rescheduled tomorrow, several times the latency — and the application layer has no idea anything changed.

★★ Problem two: STP blocks off half the bandwidth

Redundancy is mandatory — every access switch dual-homes to two aggregation switches.
But that creates a loop, and L2 networks can't tolerate loops (broadcast storms), so STP has to break it.

★ STP breaks the loop by "blocking" — it turns off the redundant link and leaves only one forwarding.
That blocked link: the light is on, the optics are up, but it forwards zero traffic.

Put it in numbers: access declares 2x40G = 80G of uplink,
after STP blocks one, only 40G is actually usable →
the oversubscription ratio goes from a nominal 6:1 to an actual 12:1 — exactly double.

You paid for that link that never forwards anything.

Try it: see both problems for yourself

Go to the architecture comparison page (the traditional tree is loaded by default):

① Look for the red dashed links on the canvas — those are STP-blocked, bought but unusable
② Use the path lookup on three pairs: Access1→Access2 (3 hops), Access1→Access3 (5 hops) — the hop counts really are different
③ Check the oversubscription column: nominal 6:1, actual 12:1

Then switch to the "standard fabric" comparison: over there every link is forwarding, and any two servers in different racks are always 3 hops apart.

Key takeaways

• The tree architecture was designed for north-south traffic; east-west traffic overwhelmed it
• ★ The fatal part isn't slowness — it's that latency depends on where the server sits — unacceptable once VMs can migrate
• ★★ STP breaks loops by "blocking" → half the uplinks are bought but unusable, doubling the oversubscription ratio
• Scaling out means touching the core, and the core is the most expensive, hardest-to-take-down device

Lesson 2 · Spine-leaf's two iron rules

Beginner · about 10 min

Goal: The entire architecture is just two rules. Understanding why they're iron rules matters more than memorizing them.

The two iron rules

① Every leaf connects to every spine.
② Spines never connect to each other, and neither do leaves.

That's it. Everything else follows from these two.

★ Corollary one: any two servers are always 3 hops apart

Starting from the two iron rules, there is only one possible path shape between two servers in different racks:

Leaf → Spine → Leaf

· Leaves don't interconnect → you can't go sideways
· Spines don't interconnect → you can never "go up twice"

★★ So no matter how big the fabric is (4 leaves or 400), the hop count is always 3.
Predictable latency — this is the single most valuable property of this architecture; distributed systems are far more sensitive to latency jitter than to absolute latency.

⚠️ Looking at it in reverse makes it clear why rule two can't be broken: the moment spines interconnect, a 4-hop leaf→spine→spine→leaf path becomes possible, and the constant hop count is gone.

★ Corollary two: uplink port count = spine count

Rule one says "every leaf connects to every spine," which means:

a leaf's uplink port count must equal the number of spines.

This isn't a coincidence — it's a hard constraint. It leads to a very practical conclusion:
★ In spine-leaf, "add a spine" and "add uplink bandwidth" are the same thing.

Conversely, however many free uplink ports a leaf has determines the maximum number of spines you can still add. You need to plan for this when buying leaves.

★ Corollary three: spine port count = the fabric's capacity ceiling

Spines don't connect to servers — every port on a spine is used to connect to leaves. So:

however many ports a spine has, that's the maximum number of leaves the whole fabric can have.

For example, a 32-port spine → at most 32 leaves → if each leaf has 48 downlink ports, that's at most 1,536 servers.

★ To break through this ceiling, you need a super-spine (multi-tier Clos) — an extra layer above the spines that connects multiple fabrics (pods) together. This is the standard approach at hyperscale.

Try it: count the links

Go to the oversubscription & scale-out page and look at the canvas below (standard fabric: 4 spines x 6 leaves):

① Count the links: it should be 4 x 6 = 24 — fully meshed, not one missing
② Look for any spine-to-spine or leaf-to-leaf link — there should be none
③ Then switch to the "wiring that violates both iron rules" scenario from the top left and see what problem it reports

★ While you're there, try "a leaf isn't fully meshed to every spine": that leaf is missing one uplink, and the oversubscription ratio immediately goes from 3:1 to 4:1 — it becomes the whole fabric's weak link, and the symptom is "only this rack occasionally drops packets."

Key takeaways

• ★ The two iron rules: every leaf connects to every spine; spines don't interconnect, neither do leaves
• ★★ Corollary: any two servers in different racks are always 3 hops apart — predictable latency
• Corollary: leaf uplink port count = spine count → adding a spine is adding bandwidth
• Corollary: spine port count = max leaf count → sets the fabric's capacity ceiling

Lesson 3 · ★ Oversubscription ratio (this module's core concept)

Advanced · about 16 min

Goal: The single most practical number in data center design. It decides, in one sentence, whether a fabric is fit to go live.

How to calculate it

Oversubscription ratio = leaf downlink bandwidth ÷ uplink bandwidth

Run it against the most common configuration in production:

Downlink: 48 x 25G  = 1200G   (to servers)
Uplink  :  4 x 100G =  400G   (to spines)
Ratio   : 1200 / 400 = 3:1

This means: if all 48 servers in this rack max out their NICs at the same time, the uplink can only carry 1/3 of that.

Why oversubscription is even allowed

It sounds like a design flaw, but it's actually deliberate, and reasonable:

★ It rests on a statistical fact: servers don't max out their NICs at the same time.
And most traffic is absorbed within the rack (instances of the same app tend to be placed together).

So oversubscription is fundamentally trading statistical multiplexing for cost — the same logic as phone networks and broadband access.
Going 1:1 non-blocking is of course the safest option, but the uplink port count and spine count both multiply several times over — for general-purpose workloads, spending that much is wasteful.

★ Which tier fits which workload

1:1     non-blocking  HPC / AI training / distributed storage
<= 3:1   healthy       ★ the mainstream value for general server pools
<= 6:1   acceptable    web/app server pools, but watch monitoring
> 6:1    risky         only when you're sure east-west traffic is light
>= 10:1  unacceptable  ★ this isn't a design, it's an incident waiting to happen

★★ Why AI / HPC must be 1:1: those jobs' east-west traffic is both heavy and synchronous (all nodes exchange gradients at the same time) — the moment one link gets congested, the slowest node drags down the whole job — the statistical-multiplexing assumption no longer holds.

★ What makes >= 10:1 hardest to diagnose is the symptom: it's not "consistently slow," it's random, intermittent latency jitter and packet loss.
Ops will suspect the application first, then the server NIC, and only at the very end think of the fabric's oversubscription ratio.

★ Theoretical vs actual ratio: which one do you design for

If a rack currently only has 24 servers plugged in, the current actual ratio is only 1.5:1 — very healthy.
But once the ports fill up (48 servers), it becomes 3:1.

★★ Design must be based on the theoretical ratio (fully populated).
Because servers get added over time — plan against the current actual ratio, and by the time the rack fills up you're already over budget, and by then what needs changing is cabling and uplink ports, which is far more expensive.

Both numbers are shown on the page — make sure to tell them apart.

Try it: run the numbers, then watch it spiral out of control

Go to the oversubscription & scale-out page:

① Check the ratio for each leaf in the standard fabric — it should be 3:1, healthy
② Switch to the "oversubscription out of control (15:1)" scenario — 48x25G downlink with only 2x40G of uplink configured — see how the health indicator flags it red
③ Then switch to "non-blocking fabric" — see the cost of 1:1 (8 spines, 400G of uplink)

★ Finally, go back to the standard fabric and take one of a leaf's uplinks down — watch its actual ratio jump from 3:1 to 4:1 immediately — the weakest-link effect.

Key takeaways

• ★ Oversubscription ratio = total downlink bandwidth ÷ total uplink bandwidth
• It's deliberate: trading statistical multiplexing for cost
• ★ <=3:1 is mainstream for general workloads; 1:1 for AI/HPC; >=10:1 is unacceptable
• ★★ Design against the theoretical (fully populated) ratio, never the current actual ratio
• Look at the worst leaf, not the average — the weakest-link effect

Lesson 4 · ★★ Add a spine or add a leaf?

Advanced · about 12 min

Goal: The single most practical lesson in this module. The two are orthogonal — mix them up and the money is wasted.

★★ Two orthogonal things

Short on bandwidth (ratio too high)  ->  add a spine
Short on capacity (ports full)       ->  add a leaf

What adding a spine does: every leaf gets one more uplink →
total uplink bandwidth grows → the ratio shrinks. Server capacity stays the same.

What adding a leaf does: one more rack →
you can connect more servers. The ratio doesn't change at all.

★★★ "Adding a leaf doesn't improve the oversubscription ratio" — the most counterintuitive part

Many people's intuition is "more devices means less congestion." Wrong.

A newly added leaf has the same up/down port ratio as the existing ones (48x25G down, 4x100G up), so its own ratio is also 3:1.

★ The result: you're just spreading the same congestion ratio across more servers, not relieving the original congestion.

Conversely, if the current ratio is already bad (say, 15:1), adding a leaf first just means more servers are affected — you have to add a spine first to bring the ratio down.

★ A third option that's often the most cost-effective: raise the uplink speed

Without adding any devices, swap the uplink from 100G to 400G:
the port count stays the same (still 4), but uplink bandwidth goes from 400G to 1600G →
the ratio goes from 3:1 to 0.75:1.

★ Why it's often the most cost-effective: it skips the rack space, power, and port budget of a new spine.
And going from 40G to 100G often only requires swapping optics — the QSFP cage is the same.

⚠️ Provided both the leaf and spine ends support that speed.

Try it: line the three options' numbers up side by side

Go to the oversubscription & scale-out page — there's a "scale-out options comparison" table on the right, with four rows side by side: current / add 1 spine / add 1 leaf / raise the uplink speed.

★★ Focus on the ratio column in the "add 1 leaf" row — it's identical to the current setup. Seeing it once is worth more than reading it ten times.

Then actually click "add a spine" and watch:
· The link count goes from 24 to 30 (the new spine connects to every leaf)
· Each leaf's uplink goes from 400G to 500G
· The ratio goes from 3:1 to 2.4:1

★ What to do once the spine runs out of ports

A spine has 32 ports → at most 32 leaves. Once that's full, there are only two ways to keep scaling:

① Replace the spines with ones that have more ports — simple, but requires swapping devices
② Add a super-spine (multi-tier Clos) — an extra layer above the spines that connects multiple fabrics (pods) together

★ Hyperscale data centers take the second route: one pod is one spine-leaf fabric, and pods interconnect through a super-spine.
The cost is that cross-pod traffic becomes 5 hops (leaf→spine→super-spine→spine→leaf) — so design tries to keep workloads that talk to each other within the same pod.

Key takeaways

• ★★ Adding a spine fixes congestion (ratio shrinks); adding a leaf fixes capacity (ratio unchanged)
• ★★★ Adding a leaf never improves the oversubscription ratio — it just spreads the same ratio across more servers
• ★ Raising the uplink speed is often the most cost-effective: no new devices, maybe just swap optics
• Spine out of ports -> replace the spines or add a super-spine (cross-pod becomes 5 hops)

Lesson 5 · ★★ ECMP and elephant flows

Troubleshooting · about 14 min

Goal: All of spine-leaf's bandwidth advantage rests on ECMP. And its one real pain point lives here too.

ECMP replaces STP

A tree architecture uses STP to break loops (blocking half the links);
spine-leaf uses L3 + ECMP — every link forwards at the same time.

★ This is the fundamental reason it can use all its bandwidth, and also why it must be an L3 fabric: ECMP is a layer-3 capability that L2 can't do (L2 can only patch around it with STP or things like MC-LAG).

★ The most ironic mistake: bought 4 spines, only 1 is actually working

ECMP doesn't just work automatically, especially with an eBGP underlay:

★ BGP, by default, only selects a single best path.
If maximum-paths isn't configured, that's the same as having no ECMP at all → out of several uplinks, only one forwards anything.

And eBGP has a second trap: by default it does a strict AS_PATH comparison, so equal-cost paths through different AS numbers aren't accepted →
you usually also need bestpath as-path multipath-relax.

⚠️ The symptom of this mistake is deceptively mild: the network is up, monitoring is green, it's just that bandwidth is only 1/4 of what was designed — and you'll never notice unless you go looking.

★★ Elephant flows: ECMP's one genuine pain point

ECMP hashes per flow (the 5-tuple: source/destination IP + source/destination port + protocol).

★ And a single TCP connection's 5-tuple is fixed →
that flow rides the same uplink for its entire life, and never uses any other link.

So a large file transfer between two servers (backups, data sync, model checkpoints) can only ever reach 1/4 of the uplink bandwidth — even if most of the fabric's total bandwidth is sitting idle.

★★ This isn't a misconfiguration — it's how ECMP fundamentally works.
And it produces a very strange symptom: one link is maxed out and dropping packets, while another link sits at 5% utilization — "there's plenty of total bandwidth, but the traffic just won't go there" is a hard thing to explain to the business side.

Three real-world ways to deal with it

① Split into multiple connections at the application layer — the most effective, and doesn't depend on any network device.
　Distributed storage systems (Ceph, HDFS) commonly do this: a transfer job opens 8 TCP connections, each with a different 5-tuple → hashed onto different links.

② Flowlet / dynamic load balancing — a chip-level solution. It slices a large flow into segments (exploiting TCP's natural burst gaps), and sends different segments over different links.
　Different vendors call it different things: DLB, Adaptive Routing, Conga.
　⚠️ The cost is that packets may arrive out of order — confirm the application can tolerate that before turning it on.

③ Raise the uplink speed — if a single link is fat enough, even one large flow fits. Simple, but costs money.

Try it: produce an elephant flow yourself

Go to the ECMP & load spreading page:

① First run 50 normal flows, 0 elephant flows — spreading is fairly even
② Then run 10 normal flows + 3 elephant flows — ★ watch the imbalance spike past 100%, with one link maxed out while another sits at 5%
③ Switch the hash mode to Flowlet and run the same flows again — the imbalance drops back to single digits

★ While you're there, switch to the "bought 4 spines, only 1 is working" scenario and watch the number of equal-cost paths go from 4 down to 1.

Key takeaways

• ★ ECMP replaces STP — every link forwards at the same time — which requires the fabric to be L3
• ★★ eBGP only selects one path by default! Without maximum-paths configured, there is no ECMP
• ★★★ Elephant flows: a single TCP connection's 5-tuple is fixed -> it only ever rides one link
• A strange symptom: one link is maxed out and dropping packets, another sits at 5% — "plenty of total bandwidth, but it won't go there"
• Fixes: split into multiple connections at the app layer (most effective) / Flowlet (can reorder packets) / raise the uplink speed

Lesson 6 · Why VXLAN + EVPN on top of all this

Advanced · about 12 min

Goal: The fabric is pure L3, but VM migration needs L2 reachability — how is that contradiction resolved?

★ Where the contradiction lies

As covered earlier: spine-leaf must be an L3 fabric, because ECMP is a layer-3 capability.

But there's a hard business requirement: a VM's IP can't change when it migrates.
And keeping the IP unchanged means the VM must stay within the same L2 domain before and after the move —
in other words, the application needs L2 reachability.

★★ Providing L2 to applications on top of a purely L3 network — that's the whole reason an overlay exists.

VXLAN: stuffing an L2 frame inside a UDP packet

The approach is straightforward: treat the entire Ethernet frame as payload and stuff it into a UDP packet (destination port 4789), route it normally across the L3 network, and unwrap it at the far end.

Outer: IP (leaf A -> leaf B) + UDP(4789) + VXLAN header
Inner: the original Ethernet frame (VM A -> VM B)

· Encapsulation/decapsulation happens on the leaf, a role called the VTEP
· The spine has no idea VXLAN exists — it only looks at the outer IP to do ECMP
★ This matters: the spine stays simple, and all the complexity is concentrated on the leaf
· VNI (24 bits) replaces the VLAN ID → expanding from 4,096 to 16 million

★ EVPN: VXLAN's control plane

VXLAN by itself only solves "how to encapsulate." There's still a question:
how does leaf A know a given MAC sits behind leaf B?

The early approach was multicast flooding (learning MACs via multicast) — it works, but it's messy: it depends on multicast, produces heavy flood traffic, and doesn't scale.

★ EVPN's approach is: use BGP to distribute MAC addresses.
(A new BGP address family, called EVPN NLRI)

Benefits:
· No more multicast — BGP advertises precisely instead
· Can advertise MAC + IP bindings → enables ARP suppression (answer locally instead of flooding)
· Supports multihoming (one server dual-homed to two leaves, identified by an ESI)

★ Worth noting: this is exactly why the fabric's underlay also uses BGP — one protocol handles both underlay and overlay, so operators only have to learn one thing.

★ The standard combination in production

underlay: eBGP (one private AS per device) + ECMP
overlay : VXLAN data plane + EVPN control plane

Why the underlay uses eBGP instead of OSPF / IS-IS (RFC 7938):
· AS_PATH naturally prevents loops, and path length directly reflects hop count
· Fine-grained policy control — to stop a leaf from forwarding, one policy statement is all it takes
· Predictable behavior — no "one flapping link triggers a full-network SPF recompute"
· Configuration is highly template-friendly, which suits automated generation — decisive at the scale of hundreds of devices

★ Worth comparing against this product's eBGP module: the AS_PATH, path-selection rules, and maximum-paths learned there all apply here too.

Try it: see what the config actually looks like

Go to the live config page and look at how a leaf's configuration splits into three sections:

① underlay: interface IPs + eBGP + maximum-paths
② overlay: VXLAN VTEP + EVPN address family
③ application: the VNI-to-VLAN mapping

★ Pay attention to how short the spine's configuration is — it only handles the underlay, and never touches VXLAN or application concerns. All the complexity lives on the leaf.

Key takeaways

• ★ The contradiction: the fabric must be L3 (for ECMP), but applications need L2 (for VM migration)
• VXLAN: stuffs an L2 frame inside UDP(4789). The VTEP lives on the leaf, and the spine is completely unaware
• 24-bit VNI -> 16 million, replacing VLAN's 4,096
• ★ EVPN: distributes MACs via BGP instead of multicast flooding; also enables ARP suppression and multihoming
• ★ The standard combination in production: eBGP underlay + VXLAN/EVPN overlay (RFC 7938)