Transport Network Evolution

10 stages, smallest to largest; each one starts from what broke in the previous version.

① Two servers

2 servers · 1 sites

This architecture: ★ One cable between two NICs, with static addresses from the same subnet at each end -- no switch, no hub, no DHCP, and not even a default gateway

What broke in the previous stage

This is the starting point: two machines need just one cable. Give them 192.168.1.1/24 and 192.168.1.2/24, ARP resolves the other side's MAC directly, and the frame is delivered.
★ Why no default gateway is needed: a gateway is "who to hand it to when the destination is on another subnet". Here there is exactly one subnet and the peer is on the same cable, so "another subnet" does not exist yet.
★ Historically this had to be a crossover cable: both ends are NICs, both transmit on pairs 1/2 and receive on 3/6, so without swapping them you get two parties talking into each other's mouth. Modern NICs support Auto-MDI-X and negotiate who transmits, so a straight cable works.
★★★ The ceiling here is very clear: it connects exactly two machines. A third has nowhere to plug in. Avoiding a device means a full mesh -- n machines need n(n-1)/2 cables and n-1 NICs each: 3 machines is 3 cables, 4 is 6, 10 is 45. That curve grows far faster than the server count, so an intermediate device becomes mandatory.

Try it（Two hosts on a cable: "same subnet" is the only condition）

★★★ Change Server 2's IP to 192.168.2.2 and it stops working immediately: the two are no longer on one subnet, so a gateway is needed -- and there is no router to ask.
★★ Or widen Server 1's mask from /24 to /16 -- now it considers the peer local again. But change only one side and you get outbound-works / reply-fails: a one-way failure.

② Three servers

3 servers · 1 sites

This architecture: You now need an intermediate device. The cheapest is a hub: it works at layer 1, repeating whatever it receives out of every other port, with no MAC lookup and no forwarding table at all. At three machines that is good enough

What broke in the previous stage

★★★ Previously one cable joined exactly two machines. Three machines without a device means a full mesh: 3 cables and 2 NICs each. Just about feasible -- but four machines needs 6 cables and 3 NICs each, and it falls apart immediately. So what really happens at this step is that a device whose only job is connecting appears for the first time.
★★ What a hub does fits in one sentence: it is an electrical repeater. Bits arriving on any port get replayed out of every other port. It has no idea what a MAC address is, makes no decisions, and therefore has no forwarding table at all.
★ The cost is that every port sits in one collision domain: only one station may talk at a time, and if two transmit at once they collide and both back off and retry (CSMA/CD). It is also half duplex -- you cannot send and receive at the same time -- and the bandwidth is shared across all ports.
★ At three machines none of that hurts yet: with only three talkers, collisions are rare. A switch would of course be better, it was just far more expensive back then. "Good enough" genuinely holds at this size -- which is why this stage is "either a hub or a switch".

Try it（Three hosts on a hub: one sends, everybody receives）

★★★ Simulate 1 -> 2 and watch Server 3 light up as well: a hub does not read MAC addresses, so whatever arrives on one port is repeated to every other. That is what a shared medium means.
★★ Move Server 2's IP to another subnet and watch the failure become an unanswered ARP -- not "no route", because routing does not exist at this scale.

③ 10 servers

10 servers · 1 sites

This architecture: A switch becomes mandatory. It works at layer 2: a MAC address table tells it which MAC lives behind which port, and it forwards only out of that port. Each port then becomes its own collision domain, can run full duplex, and gets its own bandwidth

What broke in the previous stage

★★★ A hub falls apart at ten machines, and all three consequences share one cause: every port is in one collision domain. · Collision probability climbs quickly with the number of talkers. Classic shared Ethernet delivers roughly a third to 40% of nominal throughput, and it gets worse the busier it is -- because the back-off and retransmission themselves occupy the medium; · Half duplex: send or receive, never both; · Shared bandwidth: ten machines split one 10M segment; they do not each get 10M.
★★ There is also a consequence that gets overlooked -- security: a hub copies every frame to everyone, so any machine that puts its NIC into promiscuous mode can see all the traffic.
★★★ A switch fixes all of it at once, on the strength of one capability: it understands MAC addresses. When a frame arrives on a port it records "this source MAC is behind this port" (that is learning), and when forwarding it consults the table and sends only out of the port where the destination MAC lives. Each port becomes its own collision domain -- two devices on one cable, so full duplex is possible -- and you no longer see anyone else's traffic.
★ But the switch also introduces a new finite resource: the MAC address table. It has a size; once full, new entries cannot be learned and the switch has to flood frames for unknown destination MACs out of every port -- degrading back towards hub behaviour. That comes back to bite at larger scale.

Try it（Ten hosts on a switch: only the intended port receives）

★★★ Simulate 1 -> 7 and compare with the previous scale: only Server 7 lights up and nobody else sees a byte. The switch consults its MAC table and forwards out one port.
★★ Change Server 7's mask to /28 and it is no longer on Server 1's subnet. The outbound direction still arrives (1 thinks 7 is local) but 7 cannot reply -- another one-way failure.

④ 100 servers

100 servers · 1 sites

This architecture: Split into several subnets and forward between them with a router (or an L3 switch). At the same time one switch no longer has enough ports, and several switches with redundant links create L2 loops -- so STP becomes mandatory

What broke in the previous stage

★★★ The switch solved collisions but not broadcast. Layer-2 broadcasts (ARP, DHCP, and flooding when the destination MAC is unknown) go to every station in the broadcast domain. A hundred machines means every ARP is received by 100 NICs and processed by 100 CPUs -- and broadcast volume grows with host count, so it is an O(n) problem meeting an O(n) problem.
★★★ Worse is the failure domain: one faulty NIC storming broadcasts takes all 100 machines down with it, and the switch cannot stop it -- forwarding broadcasts is precisely its job. Inside one broadcast domain, one machine's fault is everyone's fault.
★★ You also hit a physical ceiling: one switch does not have 100 ports. Once several switches are joined, you want redundant links so a single cable failure does not isolate anyone -- and a redundant link is a loop. L2 loops are fatal because the Ethernet header has no TTL: a broadcast frame circulates forever, multiplying as it goes, and saturates the switch CPU within seconds. Hence STP -- whose remedy is to block the redundant link (the grey dashed line).
★★★ The way out is to make broadcast domains smaller: several subnets, each keeping its broadcasts to itself. But then the subnets cannot reach each other -- forwarding between them requires looking at the IP address, which is a layer 3 job and beyond a switch that only reads MACs. So a router becomes mandatory here, and its value is precisely that it does not forward broadcasts.

Try it（A hundred hosts: two subnets and a router）

★★★ Simulate A -> B first: the packet must cross the router (same-subnet traffic does not). Then change the destination to an address in the same segment and watch the router disappear from the path.
★★★ Clear Segment A's gateway: the error becomes "no gateway to hand it to" and the packet is never even transmitted.
★★★ Set Segment A's gateway to 10.0.9.1: the error becomes "the gateway is off-subnet". A dead end -- reaching the gateway needs an ARP, and ARP only reaches the local segment.
★★ Widen Segment A's mask to /8: it now believes 10.0.2.11 is local, so it skips the gateway and ARPs directly -- and nothing answers. That is what an over-wide mask does.
★ Change Segment B's IP to 10.0.3.11: the error becomes "no route to that subnet", and this time the source does get an ICMP unreachable -- a router is an L3 device and can report one.

⑤ 1,000 servers

1000 servers · 1 sites

This architecture: Spine-leaf (Clos): every leaf uplinks to every spine, any two leaves are always 2 hops apart, and every link forwards simultaneously via ECMP. The prerequisite is routing all the way down to the leaf

What broke in the previous stage

★★★ At a thousand machines the previous "switches plus a router" design has two flaws it can no longer hide: · STP blocks the redundant links -- half the bandwidth you bought sits idle; · Traffic between racks must go up a layer and come back down, and that layer is deliberately oversubscribed (say 4:1). The result is that the bandwidth available between two servers depends on which two racks they happen to be in -- move the same two machines and performance changes. That is unacceptable as a design property.
★★★ Clos makes this deterministic: every leaf connects to every spine, so any two leaves are always leaf → spine → leaf, a fixed 2 hops, with as many equal-cost paths as there are spines. Need more bandwidth? Add a spine and every leaf gets wider. Need more ports? Add a leaf. That is scaling out; the previous generation scaled up -- buy a bigger core -- and eventually there is no bigger core to buy.
★★★ But there is a real prerequisite: ECMP is a layer 3 capability (hash the 5-tuple to pick among equal-cost next hops). So leaf-to-spine must be routed, not bridged -- the fabric needs an IGP or BGP inside it. "Layer 3 to the leaf" is the entry ticket; without it you are back to STP.
★ One very satisfying side effect: STP disappears from the fabric entirely. With no L2 loops there is nothing for a protocol to block.

⑥ 10,000 servers, split across data centres

10000 servers · 2 sites

This architecture: A two-tier Clos has a calculable ceiling, so add a super-spine layer and carve the fabric into pods; above that, split into multiple data centres joined by DCI

What broke in the previous stage

★★★ A two-tier Clos has a ceiling you can compute: max servers ≈ (spine port count) × (leaf downlink count). With 32-port spines and 48-port leaves: at most 32 leaves, 48 servers each, so roughly 1,500 machines. To go bigger the two-tier structure itself runs out -- you add a third tier (super-spine) and the old fabric becomes one pod.
★★★ But the limit you hit first is usually not the network: a single building has finite power, cooling and floor space. At this scale the facility itself becomes the container that cannot hold any more. So half the driver for splitting into data centres is topology and half is physics.
★★ And the failure domain: one fabric is one failure domain. A single bad config push or one control-plane bug can take ten thousand machines with it. Carving into pods and data centres is fundamentally about limiting the blast radius -- so an incident destroys one pod instead of everything.
★★★ At this scale something else becomes unavoidable: the router that faces the public internet has to appear on the diagram. · users arrive from the internet, so there must be a border; · ★★ more importantly, how does inter-site traffic travel. At this size you usually do not have a backbone of your own yet -- so either you lease a circuit from a carrier (expensive, upgrades run on their schedule) or you simply borrow the public internet: an encrypted tunnel (IPsec / GRE) between the two border routers. ★ Borrowing the internet is cheap and quick to turn up, but the cost is real: the path in the middle is decided by the internet, so you can guarantee reachability but not latency or jitter -- put database replication on it and it hurts. Which is exactly why the next scale up builds its own backbone.
★ Splitting into data centres raises a new problem: the links between them are expensive and few, compared with the bandwidth you scatter freely inside one building. So "where things live" becomes a network question for the first time -- what stays local and what is worth crossing has to be worked out.

⑦ 100,000 servers, split across cities with a backbone

100000 servers · 12 sites

This architecture: Split across cities and build your own backbone: long-haul links of your own between cities, with traffic engineering (MPLS-TE / SR-TE) to fill those expensive pipes; a hierarchical IGP; and a DWDM optical layer underneath it all

What broke in the previous stage

★★★ Until now you could assume all sites were in one metro -- tens of kilometres, under 1ms round trip, so which site hardly affected latency. Across cities the speed of light becomes a hard constraint: light in fibre travels about 200,000 km/s, so 1,000km is 5ms one way and 10ms round trip. ★★ No amount of money, bandwidth or new hardware changes that number -- it is a physical constant. So the architecture must go from "one big pool" to "geographic regions, each able to serve its own users".
★★★ Once inter-city traffic grows, buying carrier circuits stops being economical or controllable: price scales linearly with bandwidth, jitter is not yours to fix, and capacity upgrades run on someone else's schedule. So you lease or buy long-haul fibre and light it yourself -- that is where a backbone begins.
★★★ Long-haul links are expensive and few, so you cannot "just hash and scatter" the way you do inside a building. In the fabric links are cheap and hashing is good enough; on the backbone you need traffic engineering -- explicitly deciding which flow takes which path (MPLS-TE / SR-TE) -- or you end up with one pipe saturated and another idle, and that idle pipe costs millions a year.
★★ Device count also outgrows the IGP: put a few thousand devices in one area/level and every topology change forces a full SPF recomputation everywhere. So the IGP must be hierarchical (OSPF areas, ISIS levels) with BGP between regions -- limiting who needs to know about whom, which is the same idea as limiting a broadcast domain, just moved up to layer 3.
★★ Having your own backbone does not mean you can drop the public internet: users arrive from it anyway, and long-haul fibre does get cut by diggers and earthquakes -- at which point the internet is the fallback path. So the border router stays on the diagram; its role just changes from "the only path" to "the backup path".
★ At this point "the network" is no longer just routers and switches: there is an optical layer beneath (DWDM), carrying dozens of wavelengths on one fibre, and capacity planning becomes an optical question. You used to buy "a 10G circuit"; now you manage "how many wavelengths are left on this fibre".

⑧ 1,000,000 servers, intercontinental

1000000 servers · 60 sites

This architecture: Intercontinental: submarine cables, an optical layer you control yourself (ROADM, switching wavelengths directly in the optical domain), and a central controller computing paths (SR / SRv6, pushing the path into the packet header)

What broke in the previous stage

★★★ Crossing oceans is not like crossing cities, because submarine cable is a scarce resource: build time is measured in years and capacity is bought years ahead; and when one breaks (a trawler's anchor, an earthquake) repair takes weeks, not hours. So paths must be diverse by construction -- different cables, different landing stations, even opposite ways around the planet -- rather than a simple primary/backup pair.
★★★ With dozens of regions and thousands of candidate paths, steering traffic by hand is no longer possible. Hence SR / SRv6 plus a central controller: it sees global state, computes the optimum, and pushes the path into the packet header; devices in the middle just forward on labels and none of them needs a global view. ★ That is the shift from "every device computes for itself" to "one brain computes and everyone executes" -- Google's B4 and Espresso are systems of this kind.
★★ At this magnitude the optical layer has to be yours: patching a wavelength across on a ROADM is far cheaper and faster than detouring at the IP layer. ★ And a step further: packet switching at the top tier starts being replaced by optical switching -- if two pods simply need a big fixed pipe between them, an optical circuit beats pushing it through a stack of routers.
★ One last counter-intuitive but important point: at this scale most outages are self-inflicted (a config push, an upgrade), not hardware failures. So the network grows a layer of change verification and staged rollout infrastructure -- it does not appear on the topology diagram, but it is the layer that actually determines availability.

⑨ Three networks - production, management, console

10000 servers · 1 sites

This architecture: One server room and one set of devices carrying three networks at once: production for user traffic, an out-of-band management network for SSH and telemetry, and a console network over serial. ★★★ One principle runs through all of it: a management path must not depend on the thing it manages

What broke in the previous stage

★★★★ One principle runs through the page: a management path must not depend on the thing it manages. Production dies -> management is still there and you can log in to look; management dies too (or the device will not boot, or has no address yet) -> the console is still there and you can recover it; no console -> start driving. ★★ So this stage adds no scale: it fixes the scale at one large room and cuts a cross-section through three planes.
★★★★ Look at the red line in the diagram -- the most expensive lesson here. To save one cable, the management switch takes its uplink from a production switch. Fine day to day -- and when production dies, management dies with it, absent at the moment you need it. ★★★ This is the common cause behind every "we could not fix it remotely and had to drive over": the rescue path shared a segment with the thing that failed. Hence the dependency must run one way only.
★★★ Bringing a device up runs through the three in reverse -- itself the clearest proof of how they relate: enter by console to set the management address (the only path that needs no configuration at the far end), then SSH over management to push the real config, and only then configure production. Troubleshooting runs the other way. ★ In passing: a test environment has all three of its own -- another copy, not a fourth network. Whether it shares production's management network is the next stage.

⑩ Production - office, remote, test

10000 servers · 1 sites

This architecture: Office, remote work and test environments -- none of them is production, and all of them want to reach it. ★★★ This stage's axis is trust boundaries and ingress: how many doors you pass through, and what those doors themselves depend on

What broke in the previous stage

★★★ Office, remote work and test environments -- none is production, and all of them want to reach it. ★★★★ So look at the three red lines in the diagram. Each is labelled with the reason someone wired it that way -- and all three reasons sound perfectly sensible, which is exactly what makes them dangerous. There is one correct path: the VPN first, then the bastion.
★★★★ The counter-intuitive one: those two doors must not depend on the office network. ★★★ Wiring bastion login to the office AD is convenient -- but if AD is taken over you cannot get into the management network, and that is a signature step in a ransomware attack. At the moment rescue matters most, the entrance is gone. ★★ So keep local accounts plus hardware tokens on the bastion, unused day to day.
★★★ Remote work ended the assumption that the internal network is trustworthy -- an engineer's laptop shares a broadcast domain with somebody else's smart TV. ★★★★ The most immediately actionable consequence: stop treating "the source address is in an internal range" as authorisation.