Lessons (6, study in order)

Lesson 1 - What problem VXLAN actually solves

Beginner · about 8 min

Goal: Two problems: 4,094 VLANs is not enough -- and the harder one, giving two hosts an L2 adjacency across a routed fabric.

The 12-bit ceiling

A VLAN ID is only 12 bits wide in the 802.1Q tag, and once you take out the reserved 0 and 4095 you are left with 1 to 4094 usable values. In an enterprise campus that is plenty. In a multi-tenant cloud it is not: let each tenant casually ask for three or four subnets and a thousand tenants have consumed the entire space.

The worse part is that those 4,094 values are one namespace shared by the whole L2 domain -- not 4,094 per switch. If two unrelated tenants on the same fabric both want VLAN 100, you are stuck doing VLAN translation somewhere in the middle, or separating them onto different physical networks. ★ Past a certain scale a VLAN ID stops being a config knob and becomes a scarce resource that needs central allocation and bookkeeping.

VXLAN's first answer is blunt: replace the 12-bit segment identifier with a 24-bit VNI, which gives you 16 million of them. That is large enough that you can stop doing bookkeeping altogether and just derive the VNI from a tenant ID or an application ID.

The harder problem: L2 adjacency over a routed network

Running out of VLAN IDs alone would not have been enough to make the industry adopt a whole new encapsulation. The real driver is this: a modern datacenter fabric is a pure L3 spine-leaf. Every leaf uplink is a routed interface, and ECMP spreads traffic across all the spines. Why it has to be built that way is covered in detail in this product's DC Fabric module: only L3 gives you ECMP, whereas an L2 network has to block its redundant links with STP.

What the workloads want is the exact opposite. When a VM migrates between racks its IP must not change, and cluster heartbeats, VRRP and plenty of legacy broadcast-discovery mechanisms all require their members to be in the same subnet. Same IP, same subnet -- both are really asking for an L2 adjacency between those two hosts.

★ So the tension is sharp: the network underneath has to be L3 (for ECMP and scale) while the workload on top wants L2 (for mobility and shared subnets). VXLAN satisfies both at once by putting the entire L2 frame inside an L3 packet.

Overlay and underlay: two networks stacked on each other

Once you split it that way you have two logically independent networks. The underlay is the real L3 fabric, and it has exactly one job: make every leaf's VTEP address (usually a loopback /32) routable to every other one. The overlay is the set of L2 domains VXLAN manufactures for the tenants, one per VNI.

What makes this split valuable is that it separates failure domains and rates of change. The underlay routing table holds a few hundred loopbacks and transit prefixes, and then sits still for years. Meanwhile tenants create and delete subnets daily and VMs move constantly -- and none of that changes a single underlay route.

★ The flip side is that troubleshooting gets painful the moment you look at both networks as one thing. Build the habit of asking first: is this an underlay problem or an overlay problem? Can the two VTEP addresses ping each other? If they can, the underlay is fine and your problem is in the encapsulation, the VNI, or MAC learning.

Key takeaways

• A VLAN ID is 12 bits, 1-4094 usable, and it is one namespace shared by the whole L2 domain; a VNI is 24 bits, 16 million of them
• What VXLAN really solves is providing L2 adjacency over an L3 fabric -- not just the shortage of segment IDs
• The underlay only has to make VTEPs routable to each other; the overlay is what the tenant sees -- and step one of troubleshooting is always deciding which layer you are in

Lesson 2 - ★★★ The encapsulation stack: what those 50 bytes are for

Beginner · about 14 min

Goal: This module's core lesson -- inner frame -> VXLAN -> UDP -> outer IP -> outer Ethernet, all 50 bytes of it, and why each layer has to be there.

Four new headers, one job each

What a VM emits is still an ordinary Ethernet frame: destination MAC, source MAC, possibly an 802.1Q tag, then the IP packet. The VTEP does not change a single byte of it -- it just prepends four new headers:

[Outer Eth 14B][Outer IP 20B][UDP 8B][VXLAN 8B][ original Ethernet frame: dst MAC, src MAC, (802.1Q), IP ... ]
         added total = 14 + 20 + 8 + 8 = 50 bytes

Memorize that 50 bytes -- Lesson 5 is nothing but arithmetic on it. And one common confusion worth clearing up: when people say the VXLAN header is 8 bytes, they mean only the innermost of those four layers. The 50 bytes engineers quote is the sum of all four.

★ The real thing to internalize here is that none of the four layers is decoration -- each answers a specific, unavoidable question. Here is what each one is actually for.

Why each layer has to be there

The VXLAN header (8 bytes) answers which L2 domain this frame belongs to. The only parts carrying information are the 24-bit VNI and an I flag that must be set to 1; everything else is reserved. When the remote VTEP decapsulates, that VNI is the only thing telling it which L2 domain to hand the inner frame to -- the VNI is the overlay's sole identity.

The UDP header (8 bytes) answers how to get this packet across an existing L3 network unmolested. Using UDP instead of registering a new IP protocol number was a deeply pragmatic call: every router, firewall, NIC offload engine and ECMP hash implementation already understands UDP. Destination port 4789 is the IANA assignment for VXLAN. ★ Watch out for a classic trap here: early Linux implementations and some older gear default to 8472, and when the two ends disagree you get no log message at all, just no traffic.

The outer IP header (20 bytes) answers where this packet goes in the underlay: source is the ingress VTEP, destination is the egress VTEP, both leaf loopbacks. All the underlay routers see is an ordinary unicast IP packet between two loopbacks, indistinguishable from anything else they forward.

The outer Ethernet header (14 bytes) is just a normal hop-by-hop L2 header, rewritten at every hop. It has nothing to do with the inner Ethernet header that was preserved intact -- and that is where beginners get lost: a VXLAN packet carries two Ethernet headers, the outer one changing at every hop, the inner one not moving a single bit end to end.

The UDP source port is not a port, it is entropy

The one field that looks like a port but is not is the UDP source port. A VTEP never uses it to identify a session -- it is a hash of the inner packet's five-tuple (typically landing in the 49152-65535 dynamic range), and the receiver does not even look at it.

Why build it that way? Because the underlay's ECMP hash can only see the outer headers. If every VXLAN packet had identical outer headers -- same VTEP IP pair, same UDP 4789, a fixed source port -- then all traffic between two leaves would hash onto one uplink. You bought four 100G uplinks and you would be using one.

★ Translating inner-flow diversity into the one outer field that is free to vary, so the underlay can spread the load while understanding nothing about VXLAN -- that is the entire purpose of the UDP source port, and the single cleverest decision in the encapsulation design.

That also gives you a practical troubleshooting corollary: a single elephant flow between two hosts (one TCP connection) has a fixed inner five-tuple, therefore a fixed computed source port, therefore it rides exactly one uplink. VXLAN does not save you from elephant flows -- it is the same problem the DC Fabric module covers, just wearing one more header.

Key takeaways

• 50 bytes = VXLAN 8 + UDP 8 + outer IP 20 + outer Ethernet 14, and the inner frame is not modified at all
• The VNI decides which L2 domain the inner frame belongs to; UDP/4789 was chosen because every device already in production understands UDP
• The UDP source port is a hash of the inner five-tuple, purely to give the underlay ECMP some entropy -- but a single big flow still rides one link

Lesson 3 - VTEP and VNI: who encapsulates, and who has no idea

Beginner · about 10 min

Goal: All the state sits on the leaves; the spine routes on the outer IP header and never learns a VNI or an inner MAC. That asymmetry is why the underlay stays simple and scales.

A VTEP is a role, not a box

A VTEP (VXLAN Tunnel End Point) is whatever point performs encapsulation and decapsulation. In a datacenter that is normally a logical interface on a leaf switch, sourced from a loopback /32 -- but it can equally be a vSwitch inside a server (a software VTEP), a firewall, or a load balancer. ★ There is exactly one test for whether a device is a VTEP: does encap/decap happen on it.

A VTEP does two things. Inbound from the server side: take the frame off the access port, look up which VNI that port or VLAN maps to, look up which remote VTEP holds the inner destination MAC, and build the four headers from the previous lesson. Outbound from the fabric side: receive a UDP 4789 packet addressed to its own VTEP address, strip the outer headers, use the VNI to find the matching local L2 domain, and forward the inner frame as ordinary L2 traffic.

So a VTEP needs two tables: VNI to local VLAN/port mappings, which are locally configured, and inner MAC to remote VTEP IP, which is either learned by flooding or advertised by EVPN. Where that second table comes from is the subject of Lesson 6 and of the whole EVPN module.

The spine has no idea -- and that is a design goal, not an accident

What does the spine in the middle receive? An ordinary unicast IP packet between two loopback addresses. It looks up the destination in its routing table, picks a next hop, forwards, done. ★ It never parses the UDP payload, so it does not know the VNI, does not know the inner MACs or inner IPs, and has no idea how many tenants exist on this fabric.

That ignorance buys real engineering value. The spine's MAC table holds only its underlay neighbors, so it does not grow as tenant VM counts grow. A tenant creating a subnet or migrating a VM requires zero config or table changes on the spine. The spine's role therefore collapses into one thing -- a fast L3 forwarding box -- which means you can buy the cheap, port-dense model without paying for large table scale.

The mirror image is that all the complexity concentrates on the leaves: VNI mappings, MAC learning, BUM handling, ARP suppression, all of it. That is a deliberate trade: there are many leaves and each handles few tenants, while there are few spines and every packet crosses one. Keep the state on the leaves and the design scales.

Both ends must agree on the VNI, and nothing will warn you

An overlay L2 domain only works if both VTEPs are configured with the same VNI. If leaf-1 maps its server port to VNI 10010 while leaf-3 maps its to 10011, then packets leaf-1 encapsulates arrive at leaf-3, leaf-3 looks up L2 domain 10011 -- and either finds nothing and drops the frame, or worse, lands squarely in another tenant's domain.

What makes this failure nasty is that it looks exactly like a healthy fabric: underlay routing is fine, the two VTEP addresses ping each other, every interface is up, the logs are clean. You can only find it by diffing the two configs or reading the VNI table. ★ That is what the vni-mismatch scenario loaded right now is here to teach -- being able to ping the remote VTEP does not mean the overlay works.

Look at the topology page with this scenario loaded, then switch to baseline and look again: the two pictures are identical, because at the underlay level they are identical. Every trap in this module has that same shape -- the thing that is actually broken is not visible in the drawing.

Key takeaways

• A VTEP is wherever encap/decap happens, usually a loopback on a leaf -- a spine is not a VTEP
• The spine only reads the outer IP header, so it never learns a VNI, an inner MAC or a tenant count, and its tables do not grow with tenant scale
• Both ends must agree on the VNI; a mismatch leaves the topology, the interfaces and the VTEP pings all looking perfectly healthy

Lesson 4 - BUM traffic: multicast group or ingress replication

Advanced · about 12 min

Goal: The overlay has no native flood action, so somebody has to make the copies. Two designs, two very different bills.

Why BUM needs a design of its own

BUM is shorthand for three traffic types: Broadcast (ARP requests, DHCP Discover), Unknown unicast (a destination MAC not yet learned), and Multicast. An ordinary L2 switch treats all three identically -- send it out every port in that VLAN except the one it arrived on. That is flooding.

But VXLAN encapsulation is a point-to-point unicast tunnel: the outer IP header has exactly one slot for a destination VTEP address. ★ In other words the overlay has no native flood primitive -- a frame that must reach every remote VTEP in a VNI either gets duplicated by the ingress VTEP itself, or gets handed to the underlay to duplicate.

That is why BUM needs its own design, and the fork in the road is precisely this: who does the replicating -- the underlay (multicast) or the ingress leaf (ingress replication).

Option one: a multicast group per VNI

Assign each VNI an underlay multicast group -- say VNI 10010 maps to 239.1.1.10. Every VTEP configured with that VNI joins the group. When the ingress VTEP has a BUM frame, it simply puts the group address in the outer destination IP and sends exactly one copy.

The underlay multicast tree does the replicating, and copies are only made at branch points, so no link ever carries a duplicate. This is the bandwidth-optimal answer: no matter how many VTEPs are in the VNI, the ingress leaf's uplink carries one copy.

The price is that your underlay now runs multicast: PIM, an RP, multicast routing state, an address plan of one group per VNI, and multicast troubleshooting. ★ The reason production networks largely walked away from this is unglamorous -- maintaining an entire multicast control plane to support an internal helper function is a terrible ratio of operational complexity to benefit, and multicast faults are an order of magnitude harder to chase than unicast ones.

Option two: ingress replication, and how to price it

Ingress replication (also called head-end replication) has the ingress VTEP make N-1 copies of the BUM frame itself, where N is the number of VTEPs in that VNI. Each copy gets a different destination VTEP address in the outer header and goes out as plain unicast. The underlay never has to know what multicast is.

Price this honestly. Say a VNI spans 8 VTEPs and one host sends a single ARP broadcast: the ingress leaf generates 7 packets, and its uplinks carry 7x that traffic. The more VTEPs in the VNI, the bigger the multiplier. ★ Ingress replication costs O(N), and the entire bill is paid by one leaf's uplink bandwidth and forwarding ASIC.

So why does production pick it anyway? Two reasons. First, a unicast-only underlay is simple to configure and far easier to debug. Second -- and this is the causal chain this module most wants you to keep -- ★ once EVPN is in place, the largest component of BUM (unknown unicast) mostly disappears: MACs are advertised by BGP before any traffic flows, so nothing is unknown, and ARP can be answered locally by the leaf (ARP suppression). What is left is genuine broadcast, a small enough volume that 7x stops mattering.

The choice was never about which technique is more advanced. It is about whether you would rather carry multicast in the underlay or EVPN in the control plane. The industry picked EVPN.

Key takeaways

• BUM = broadcast + unknown unicast + multicast; VXLAN is a point-to-point unicast tunnel, so the overlay has no native flood action
• Multicast groups are bandwidth-optimal but require PIM and an RP in the underlay; ingress replication needs no multicast but the ingress VTEP builds N-1 copies itself
• Ingress replication is O(N) and the whole bill lands on one leaf -- it is only acceptable because EVPN removes the unknown-unicast share of BUM

Lesson 5 - ★★★ Troubleshooting: ping works, file transfers hang

Troubleshooting · about 14 min

Goal: Partial success is the signature of an MTU problem. Learn to go from the symptom straight to the number you have to configure on the underlay.

The symptom: why it is always small packets fine, big packets dead

The report from the field usually sounds like this: two VMs ping each other perfectly, you can ssh in and type commands fine, but scp a large file and it hangs; or a web page loads halfway and stalls; or the database connects and then times out on any large query.

This flavour of partial success points almost straight at MTU, because packet size correlates strongly with the kind of traffic. A ping is 64 bytes by default, interactive ssh keystrokes are tens of bytes, a TCP handshake is small -- all far below any MTU, all sail through. A bulk transfer or a large query response fills the MSS and hits the wire at full length immediately.

★ So build the reflex: whenever connectivity is fine but bulk traffic is not, check MTU before you go read routes and ACLs. The converse is just as useful -- if ping itself fails, MTU is almost certainly not your problem.

There is a subtler mechanism underneath. The MSS a host negotiates during the TCP handshake is computed from its own NIC MTU (1500 - 40 = 1460), and the host has no idea 50 more bytes will be strapped on in transit. So it confidently emits a 1500-byte frame, which becomes 1550 after encapsulation, and the underlay interface with its 1500-byte MTU drops it.

The arithmetic: what to set the underlay MTU to

The formula is just the 50 bytes from Lesson 2: underlay interface MTU >= largest inner frame + 50. With a standard 1500-byte inner IP MTU, the underlay needs at least 1550. If the inner frame also carries an 802.1Q tag (4 bytes), it is +54, so 1554.

One thing trips people up constantly: vendors do not agree on what an interface MTU counts. Some count the L3 payload (IP MTU), some count the whole frame including the Ethernet header (sometimes called baby giant or frame size). ★ So do not memorize 1550 -- memorize the plus 50 relationship, convert it using your own platform's definition, then verify by measurement.

The measurement is a ping with the DF bit set and an explicit size. From the host, 1472 bytes of payload (1472 + 20 IP + 8 ICMP = 1500) should pass while 1473 fails if the path MTU is 1500. Bisect from there and you can pin down the exact hop that is choking.

Why the standard fix is jumbo frames on the underlay

There are three theoretical fixes. (1) Lower every host MTU from 1500 to 1450: technically valid, but you would have to touch thousands of VM, container and bare-metal NIC configs and then guarantee that nothing built in future ever misses it -- operationally a non-starter. (2) Have the VTEP fragment and reassemble overlay packets: expensive, unsupported on a lot of hardware, and fragmentation is its own source of pain. (3) Raise the underlay MTU.

★ The industry answer is (3): turn on jumbo frames uniformly on every underlay hop, typically 9000 or 9216. Why so large rather than exactly 1554? Because it settles the question forever -- add QinQ later, add MPLS, switch to a longer encapsulation, let the inner workload run 9000-byte jumbos, and you never have to come back and redo it. The configuration effort is identical and the headroom is free.

The words that matter are every hop. Miss a single interface on the underlay path and the symptom appears in full -- but only for traffic that crosses that hop. And since the fabric is ECMP multipath to begin with, you get something genuinely baffling: between the same host pair, some connections work and some do not, depending on which uplink the hash picked. That intermittent pattern is the classic fingerprint of one missed interface.

Key takeaways

• Ping works but bulk transfers hang is the signature MTU symptom: small packets slip under the limit, full-size ones do not
• Underlay MTU must be at least inner + 50, or +54 if the inner frame carries an 802.1Q tag; memorize the plus 50, not the number 1550
• The standard fix is 9000/9216 jumbo frames on every underlay hop; the fingerprint of one missed hop is some connections working and some not between the same host pair

Lesson 6 - Troubleshooting: the ceiling of flood-and-learn (and the door into EVPN)

Troubleshooting · about 12 min

Goal: With only the VXLAN data plane, MAC locations are guessed from traffic. It works, and it does not scale -- which is exactly why the EVPN module exists.

With no control plane, where does the MAC table come from

Every lesson so far has dodged one question: how does the ingress VTEP know which remote VTEP holds a given inner destination MAC? With only the VXLAN data plane, the answer is that it does not know at first -- it has to ask by flooding and then learn by peeking.

The full sequence: host A sends to host B, leaf-1 has no entry for B's MAC, so this is an unknown unicast and gets flooded per Lesson 4 (N-1 copies, or into the multicast group) to every remote VTEP in the VNI. Each of them decapsulates and floods locally, and only the leaf-3 side, where B lives, gets an answer.

The crucial part is that learning happens at the moment of decapsulation: when leaf-3 receives that packet it sees the outer source VTEP IP and the inner source MAC at the same time, so it records that MAC-A lives behind VTEP-1. When B replies, leaf-3 already knows how to encapsulate, and leaf-1 learns MAC-B behind VTEP-3 from that reply. ★ That is flood-and-learn: the data plane serving as its own control plane -- nothing was ever advertised by any protocol, every entry was inferred from real production traffic.

Three places where it stops scaling

(1) Every new destination costs one flood. A new VM coming up, or any pair of services that has not talked before, guarantees the first packet is an unknown unicast and gets replicated N-1 times across the whole VNI. In an environment with many VTEPs per VNI and churny workloads, that background noise is permanent.

(2) Aging makes you pay again, on a cycle. MAC entries have an aging timer, commonly 300 seconds. Any pair of hosts that stays quiet longer than that loses its entry, and the next conversation triggers the whole flood-and-learn again. So this is not a startup transient -- it is steady-state flooding.

(3) You can only wait passively; you cannot query. A VTEP's table only holds entries for traffic it has actually seen. Two other hosts in the same VNI can be chatting at full tilt and the local VTEP knows nothing about them. ★ Two very practical consequences: nowhere in the whole network holds a full picture of that L2 domain (during troubleshooting each leaf shows you only a partial truth), and after a VM migrates, the stale entry at the old location persists until it ages out or new traffic overwrites it -- and until then, traffic is being sent to the wrong VTEP.

One more layer: ARP requests are broadcasts, so under flood-and-learn every single ARP does a full VNI-wide replication. In an L2 domain with a few thousand VMs, ARP alone generates a meaningful background load.

Hands-on: see what one unknown unicast really costs

The flood-and-learn scenario loaded right now has every VTEP's MAC table cleared -- exactly the state you are in after aging. Go to the BUM page, send a unicast to an unknown MAC, count how many copies get generated, and look at what each VTEP learned from that one packet.

Then do the exact same thing again. The second time there is no flood, because the entries are learned. ★ Notice how enormous the gap is: for the same host pair and the same application, the first packet and every packet after it are not even in the same cost bracket -- and when the aging timer expires, you are back at the first one.

That also explains a very common field complaint: an application is a bit slow at first and then fine, and packet captures show periodic broadcast spikes on the fabric. Those are two faces of the same mechanism.

The handoff: this is precisely what EVPN fixes

Put those three limits together and the conclusion is clean: flood-and-learn works, and should not be scaled. Every one of its weaknesses traces back to the same root cause -- MAC location can only be inferred after the fact from the data plane, because nobody told you in advance.

★ So the missing piece is not a better encapsulation, it is a control plane: a protocol that advertises which MAC sits behind which VTEP to every interested VTEP before any traffic flows. That is the entire subject of the sibling module in this category, EVPN -- using MP-BGP's L2VPN EVPN address family to distribute MACs and MAC/IP bindings, so that unknown unicast largely disappears, ARP can be answered locally by the leaf, and a VM migration can actively withdraw and re-advertise.

The recommended order is exactly the one you are on: get the data plane solid here in VXLAN -- encapsulation, VTEP, VNI, BUM, MTU -- then go to the EVPN module for how the control plane removes the flooding entirely. The two modules are two halves of the same network, and either half on its own always feels like something is missing.

Key takeaways

• Flood-and-learn = the data plane acting as its own control plane: flood the unknown unicast, then learn from the outer source VTEP plus the inner source MAC on decap
• Three ceilings: every new destination costs one flood, aging makes that cost recur forever, and every VTEP only ever has a partial view
• The root cause is that MAC location can only be inferred after the fact -- adding a control plane that advertises MACs in advance is the entire subject of the EVPN module