Evolution notes for this page (click to expand)

Data Center Architecture Evolution

8 stages, smallest to largest; each one starts from what broke in the previous version.

① Two servers

2 servers · 1 sites

This architecture: One switch; only web faces the internet (80/443) while the database has a private address only -- the two talk to each other over the private network

What broke in the previous stage

  • This is the starting point: a video site just went live, one box runs web, one runs the database.
  • ★ The database has a private address only and talks to web over the private network; it is not reachable from the internet at all. This is something you should get right at two machines, not once you have scale -- the only thing exposed is web's 80/443.
  • ★ The real problem is that there is no redundancy at all: the switch, the broadband line and each server are single points of failure. You get away with it only because almost nobody notices when it breaks.
  • ★ Also there is nothing in front of web: access control is whatever iptables rules live on that one host. The next step moves that into a dedicated device.

Try it（web and the database: one private network）

  • ★★ Move the database to another subnet and watch it break: there is no router at this scale, so "different subnet" simply means "unreachable".

② Three servers

3 servers · 1 sites

This architecture: Add a second web server, so for the first time you can take one out of service and still serve; and move the perimeter into a firewall / NAT so north-south policy lives in one place

What broke in the previous stage

  • ★★★ Previously there was only one web server: one reboot, one deploy, one crash and the service was down, with no option to take it out of rotation. So the third machine is not a new role -- it is a second web server.
  • ★★ At the same time one web server can no longer carry the egress bandwidth and concurrent connections. This is especially true for video: bandwidth grows much faster than CPU, so adding CPU does not help.
  • ★ Two web servers immediately raise a new question: how do requests get spread across them. At this size a load balancer is not worth it yet, so DNS round-robin does the job -- but it has a catch: DNS is cached, so pulling a broken server out does not take effect immediately and some users keep hitting it. That catch is only really fixed at ten servers, with a load balancer.
  • ★ The firewall / NAT is not here to "hide the database" -- it was already on the private network in the previous stage. It is here to move access control from a separate iptables ruleset on every host into one place: the more machines you have, the more host-local rules drift apart.

Try it（Two web servers and a database）

  • ★★ Both web servers must reach the database. Break one of their masks and you get "half the users are fine, half get errors" -- the hardest kind of fault to chase.

③ 10 servers

10 servers · 1 sites

This architecture: One rack with a top-of-rack switch; VLANs split web / app / database into three segments; a load balancer goes in front

What broke in the previous stage

  • ★★ Previously every machine sat in one broadcast domain. At ten machines that starts to hurt: any broadcast (ARP, DHCP, a misbehaving NIC) has to be processed by everyone.
  • ★★ And one web server can no longer carry the traffic: for video, egress bandwidth grows much faster than CPU, so requests must be spread across machines. Hence the load balancer.
  • ★ Segmentation is also a security need: if web is compromised it should not be able to reach the database directly. The firewall handles north-south; VLANs handle east-west.

Try it（Ten hosts in three VLANs: crossing a VLAN needs the L3 switch）

  • ★★★ web 1 -> database 1 crosses VLANs, so it must go up to the ToR's L3 interface and back down. Compare with web 1 -> web 2: same VLAN, bridged directly, and the L3 interface is never involved.
  • ★★★ Clear web 1's gateway: same-VLAN traffic still works, everything cross-VLAN dies. ★ Which explains the classic symptom -- "I can ping my own subnet but nothing else" is almost always the gateway.
  • ★★ Give database 1 an address from 10.0.20.x (the app subnet): the arithmetic works but the cabling still says VLAN 30, so you get "delivered to the segment, but nobody is there". That is what addressing and wiring disagreeing looks like.

④ 100 servers

100 servers · 1 sites

This architecture: The classic access-aggregation-core three-tier design; dual uplinks per ToR; VRRP for gateway redundancy; a separate out-of-band management network

What broke in the previous stage

  • ★★★ Previously one ToR failure took out the whole rack. A hundred servers do not fit in one rack, so now there are several; and once there are several racks you need a layer to join them -- aggregation.
  • ★★ A single switch has also run out of ports. That is the most mundane physical constraint, and it is the direct reason the three-tier design exists.
  • ★★★ Redundancy brings loops, so STP becomes mandatory -- and STP's answer is to block the redundant links. The two grey dashed lines are bandwidth you paid for and cannot use. You carry that cost until leaf-spine arrives.
  • ★ One more thing becomes mandatory here: an out-of-band management network. Misconfigure a switch so it locks out its own management port and, without OOB, your only option is driving to the site with a serial cable.

Try it（A hundred hosts: the gateway lives on the aggregation layer）

  • ★★★ Note the gateway lives on the aggregation layer, not the ToR -- so cross-VLAN traffic has to go up to aggregation and back down. That is the cost of inter-rack traffic in the three-tier design, and exactly what leaf-spine later fixes.
  • ★★ Point the VLAN 20 servers' gateway at 10.1.10.1 (the other segment's gateway) and you get "the gateway is off-subnet". ★ A very easy mistake once you have many VLANs.

⑤ 1,000 servers across 2 data centres

1000 servers · 2 sites

This architecture: Two mutually-backing data centres; MLAG replaces STP blocking; the company gets its own AS number and IP prefixes and multi-homes with BGP; DNS/GSLB steers users between sites

What broke in the previous stage

  • ★★★ Previously the entire site was one failure domain. A power cut, a fibre cut by a digger, a false fire-suppression trigger -- any one of them stopped everything. Device-level redundancy is not enough; you need site-level redundancy.
  • ★★★ The STP bill finally comes due: half the uplinks are blocked, so half the bandwidth you bought is thrown away. MLAG -- making two switches look like one to the devices below -- lets both uplinks forward at the same time.
  • ★★ Something else changes: the IP addresses used to come from the carrier, so switching carriers meant renumbering. Now you need your own AS number and prefixes and multi-home to two carriers with BGP -- if one fails, routing converges away and users change nothing.
  • ★ Once both sites are live a new question appears: which one should a user go to. That is where DNS/GSLB comes in.

⑥ 10,000 servers across 4 data centres in 2 cities

10000 servers · 4 sites

This architecture: Leaf-spine (Clos) replaces the three-tier design and every link forwards via ECMP; VXLAN + EVPN provide multi-tenancy and workload mobility; metro sites are joined by dark fibre/DWDM and the two cities by an owned backbone; services are announced with anycast

What broke in the previous stage

  • ★★★ The three-tier design assumed traffic was north-south (user to server). But video transcoding, distributed storage and analytics make east-west (server to server) traffic dominant, and east-west has to cross the aggregation layer -- a layer deliberately oversubscribed and never designed for it.
  • ★★★ There are only 4,094 VLANs. Once machines get carved up per business line -- and later per tenant -- that number runs out fast. VXLAN's VNI space holds 16 million.
  • ★★ Virtual machines must migrate across racks without renumbering. In the three-tier design the gateway lives on the aggregation layer, so a new rack means a new subnet; EVPN's distributed anycast gateway makes every leaf the same gateway.
  • ★ Two sites are no longer enough: metro active-active survives a site failure but not a whole-city event (an earthquake, a regional blackout, a city-wide fibre cut). Hence two cities with two sites each.

⑦ 100,000 servers across data centres in several countries

100000 servers · 12 sites

This architecture: An owned global backbone with traffic engineering (MPLS / SR-TE); direct peering at IXPs in many places to buy less transit; CDN PoPs pushed as close to users as possible; submarine cables between continents; the control plane split per region

What broke in the previous stage

  • ★★★ Previously inter-city traffic rode purchased transit. Once you cross borders that stops working: international transit is expensive and jittery, and video is acutely sensitive to loss and jitter. So you buy or lease trans-oceanic capacity and make the backbone your own.
  • ★★★ More fundamentally, physical distance: a user in Europe hitting a data centre in Asia pays 250ms per round trip, so playback starts slowly and seeking stalls. Adding bandwidth cannot fix this -- light does not go faster. The only answer is to move content next to the user, which is what a CDN PoP is.
  • ★★ Past a certain volume, paying carriers stops making sense: peer directly with the other side at an IXP and much of that traffic costs nothing, takes one hop less and behaves better. You move from buying bandwidth to swapping it.
  • ★ And one thing that is not technical at all: data must stay inside national borders. Compliance dictates where the data centres get built.

⑧ 1,000,000 servers across purpose-built DCs, PoPs, cloud and colocation

1000000 servers · 60 sites

This architecture: Data centres specialise: hyperscale core sites (compute, storage, AI training, with lossless RDMA inside), regional sites, edge PoPs, DDoS scrubbing centres, and neutral colocation for interconnect; the backbone runs SR/SRv6 under an SDN controller; beneath it all sits an OTN/ROADM optical layer; and the company now sells cloud, so tenant isolation and private-link access become products

What broke in the previous stage

  • ★★★ Until now every site was the same kind of site. At this scale that breaks in three directions at once: · AI training needs a lossless, ultra-low-latency RDMA fabric, nothing like a site serving web pages; · the edge needs proximity to users, not compute; · interconnecting with others has to happen in neutral facilities, not your own. One kind of site doing everything is wrong on both cost and latency.
  • ★★★ Once you sell cloud, the network itself becomes the product: customers want their own isolated network (a VPC), a private circuit from their own premises, and multi-cloud reach. Tenant isolation used to be internal tidiness; now it is an SLA you are contractually on the hook for.
  • ★★ Attack traffic reaches another order of magnitude too: a few hundred Gbps of DDoS saturates the ingress outright. You need dedicated scrubbing centres that traffic is diverted to, cleaned and re-injected -- you cannot do that inside a production site.
  • ★ Hand-steering the backbone is no longer possible (thousands of paths). Hence SR/SRv6 plus a central controller that computes global optima and pushes the path into the packet header -- the same problem Google's B4 and Espresso were built to solve.