Lesson text for this page (click to expand)

EVPN Control Plane

Lessons (6, study in order)

Lesson 1 - What problem EVPN actually solves

Beginner · about 10 min

Goal: First be clear about why VXLAN on its own does not hold up, then introduce EVPN as a control plane -- data plane and control plane each do exactly one job.

Recap: VXLAN on its own can only flood and learn

The VXLAN module (the other tab in this category) is about encapsulation: how an L2 frame gets stuffed into UDP 4789 and crosses a pure L3 fabric behind an outer IP header. But encapsulation does not answer one question -- how does leaf-1 know that some MAC sits behind leaf-3? That information is not in the VXLAN header, and the VXLAN spec does not say who is supposed to provide it.

With nobody to provide it, a VTEP has exactly one option left: flood and learn. A destination MAC that is not in the table is treated as unknown unicast and, together with ARP, DHCP and every other broadcast, replicated to every remote VTEP in that VNI (underlay multicast, or head-end replication into N-1 unicast copies). Whoever has that host behind it replies, and the VTEP infers from the reply's outer source IP that the MAC lives behind that VTEP, then writes it down.

What flooding costs, and where the real problem is

This works. The problem is what it costs. First, amplification: with 40 leaves in the fabric, one unknown unicast is 39 copies, and ARP in a large L2 domain never stops. Second, the table comes from the data plane: the MAC table is a by-product of user traffic, so it is only as accurate as the last packet -- a host that never speaks on its own (a cluster VIP that only ever answers is the classic case) is simply unreachable until somebody ARPs for it. Third, convergence waits on aging: move a VM from leaf-1 to leaf-3 and the other leaves keep sending to leaf-1 until the entry times out, which is minutes of black-holing.

★ So the real problem is not that flooding does not work -- it is that the cost of being correct grows with both the number of leaves and the rate of change, and the fabric's control state is a side effect of user traffic. Neither is acceptable once the fabric gets big.

EVPN is a control plane -- know first, forward second

EVPN's answer is to hand the question of who is where to a routing protocol. It is an MP-BGP address family (AFI 25 L2VPN / SAFI 70 EVPN) and it inherits everything BGP already has: TCP sessions, incremental updates, route reflectors, policy, authentication. Every leaf advertises the MACs it learned locally as BGP routes, and every other leaf installs them straight into forwarding.

That is a reversal of direction. Flood-and-learn infers control information from the data plane; EVPN gets the information from the control plane and then forwards. The consequence is that the MAC table is already populated before the first packet arrives -- unknown unicast becomes a rare event, and plenty of production fabrics simply turn unknown-unicast flooding off entirely.

★ The same reversal fixes all three costs from the previous step: no more copying to everybody (N-1 copies become one BGP update); the table no longer depends on whether a host has spoken (a silent host can still be advertised); and a change no longer waits on aging (a host leaving is a withdraw, an explicit deletion).

There is one more thing that is worth real money operationally: underlay and overlay run the same protocol. You already deployed BGP to route the fabric, you already have route reflectors, monitoring and a runbook -- the overlay is just one more address family on the sessions you already have.

Name the split between data plane and control plane

The sentence to take away from this lesson: VXLAN is the data plane, EVPN is the control plane, and the two are orthogonal. VXLAN only defines what a frame looks like and how it is encapsulated and decapsulated. EVPN only defines how reachability is distributed. VXLAN knows nothing about any remote host; EVPN forwards not a single packet.

Orthogonal means either side can be swapped out. VXLAN can run without EVPN, falling back to multicast flood-and-learn (that is exactly what a lot of older deployments do). EVPN can run without VXLAN -- it was originally defined for MPLS (RFC 7432), VXLAN encapsulation was bolted on later (RFC 8365), and the same EVPN control plane rides over MPLS or SRv6 today. So in the spelling EVPN/VXLAN, read that slash as control plane / data plane.

★ During an incident this sentence decides where you start looking: drops, MTU, uneven ECMP, bad encapsulation are data-plane problems, so go look at VXLAN and the underlay; one host unreachable, one VNI dead, a black hole after a migration are control-plane problems, so go look at whether the BGP EVPN route was received and whether it was imported. Investigating both classes at once is the most reliable way to waste an afternoon.

Go to the '② Fabric' page and look at this fabric: the spines carry underlay routes and reflect EVPN routes, but a spine is not a VTEP -- it neither encapsulates nor decapsulates. Every VTEP and every EVI lives on a leaf.

Key takeaways

  • VXLAN only defines encapsulation; on its own it can only flood and learn, and the cost grows with both leaf count and rate of change
  • EVPN is an MP-BGP address family that hands every leaf the MACs before the first packet arrives
  • Data plane (VXLAN) and control plane (EVPN) are orthogonal: VXLAN runs without EVPN, and EVPN runs over MPLS or SRv6

Lesson 2 - ★★★ The five route types

Beginner · about 16 min

Goal: The lesson you will come back to most -- what each of the five route types carries, who sends it, who consumes it, and what breaks without it.

Why it is split into five types

The very first field of an EVPN NLRI is the route type. Five completely different kinds of information in one address family is not historical baggage, it is deliberate: L2 reachability, L3 reachability, how BUM gets replicated, and how multi-homed leaves coordinate are four unrelated problems, and merging them into one route type would force every box to parse things it does not care about.

Build the cheat sheet first, because this is what you will keep coming back to:

- RT-1 Ethernet Auto-Discovery: which Ethernet segment I am attached to (multi-homing auto-discovery)
- RT-2 MAC/IP Advertisement: this MAC (optionally with an IP) is behind me (the workhorse)
- RT-3 Inclusive Multicast: send me the BUM traffic for this VNI
- RT-4 Ethernet Segment: who I share an Ethernet segment with (the input to DF election)
- RT-5 IP Prefix: this IP prefix is behind me

★ The counts are wildly lopsided: in a healthy fabric RT-2 is usually over 95% of the EVPN table, RT-3 is one route per leaf per VNI, RT-1 and RT-4 only exist if you configured multi-homing, and RT-5 only exists when you need to advertise a prefix rather than a host. The failure rate runs the other way -- the hardest problems tend to live in those few sparse routes.

RT-2: one route does three jobs

RT-2 MAC/IP Advertisement -- the workhorse. Who sends it: the leaf where the host was learned locally. What it carries: the MAC, optionally an IP (that is, the ARP/ND binding), the L2VNI, optionally the L3VNI and Router MAC, the ESI, and a MAC mobility sequence number. Who consumes it: every leaf whose EVI imports the matching RT. What breaks without it: unicast reachability, full stop -- you are back to flood-and-learn. And because there is no IP-to-MAC binding, the ARP suppression cache is empty, so ARP has to flood too. One RT-2 feeds the L2 forwarding table, the L3 host route and the ARP cache at the same time, which is why it does the work of three routes.

RT-3: it builds the BUM replication list

RT-3 Inclusive Multicast Ethernet Tag. Who sends it: every leaf that has the VNI configured locally (one route per leaf per VNI). What it carries: the VNI, my tunnel endpoint address, and the tunnel type (ingress replication, or underlay multicast). Who consumes it: every leaf in that VNI -- they assemble these routes into the BUM replication list. What breaks without it: the replication list is empty, so BUM traffic (broadcast, unknown unicast, multicast) is dropped rather than flooded. The symptom is the most misleading of the lot: unicast works fine, but DHCP never gets an address, unsuppressed ARP fails, and multicast receivers get nothing -- it looks like only some traffic is broken.

A fair question at this point: if EVPN distributes every MAC, why is flooding still needed at all? Because some traffic is always outside what EVPN covers -- a host's very first ARP (if suppression is off), DHCP discover, non-IP protocols, real multicast, and any MAC that has not been advertised yet. EVPN demotes flooding from the normal case to the exception, it does not abolish it. ★ That is why RT-3 is mandatory infrastructure for every VNI rather than an optional feature.

RT-4 and RT-1: what multi-homing needs

RT-4 Ethernet Segment. Who sends it: every leaf attached to an Ethernet segment, one route each, carrying the ESI plus its own originating IP, scoped to just the neighbors on that segment by a dedicated ES-import RT. Who consumes it: the other leaves on the same segment. Why it exists: it is the one and only input to DF election. What breaks without it: each leaf concludes it owns the segment alone, so each elects itself DF, and the multi-homed host receives every BUM frame twice -- that is lesson 6's trap.

RT-1 Ethernet Auto-Discovery comes at two granularities. The per-ES route carries the ESI label and the single-active / all-active flag, and it drives split-horizon and fast convergence -- when the link to the segment fails, one per-ES withdrawal replaces what would otherwise be tens of thousands of per-MAC withdrawals. The per-EVI route implements aliasing: it tells remote leaves that this ESI is reachable through me as well. Without it, the host's MAC is only learned on whichever leaf the server hashed onto, remote leaves send 100% of the traffic to that one leaf, and half the LAG you paid for sits idle.

RT-5: prefixes, and how the five types divide the work

RT-5 IP Prefix. Who sends it: a leaf or border device that needs to advertise a prefix rather than a host. When you need it: external and default routes (toward a firewall or an internet edge), summarized subnets, networks that live outside the fabric, and silent subnets where no host route will ever be learned. Who consumes it: the corresponding L3VNI / IP-VRF. What breaks without it: host routes carried in RT-2 still work inside the fabric, but any destination that is not a locally ARP-learned host -- summaries, external prefixes, the default route -- has no way to get in.

★ One sentence that ties the five together: RT-2 provides reachability, RT-3 is what makes flooding possible at all, RT-1 and RT-4 exist only because of multi-homing, and RT-5 exists because not every destination is a host you learned by ARP.

Hands-on: count the route table by type

Go to the '③ EVPN route table' page and step the filter through the route types. Count the RT-2 routes, check that RT-3 really is exactly one per leaf per VNI, then confirm that RT-1 and RT-4 only appear on the two leaves that have an ESI configured.

Then expand one RT-2 and line its fields up against this lesson's cheat sheet: MAC, IP, L2VNI, L3VNI, Router MAC, ESI, sequence number. In particular convince yourself that one route is serving both L2 forwarding and the L3 host route -- that fact is the prerequisite for symmetric IRB in lesson 5.

Key takeaways

  • RT-2 is the workhorse: MAC, optional IP, L2VNI/L3VNI, ESI, mobility sequence -- one route feeds L2 forwarding, the L3 host route and the ARP cache
  • RT-3 builds the BUM replication list; without it BUM is dropped rather than flooded, and the symptom looks like only some traffic being broken
  • RT-1 and RT-4 exist only for multi-homing (RT-4 is the input to DF election, RT-1 gives split-horizon and aliasing), and RT-5 advertises prefixes rather than hosts

Lesson 3 - RD vs RT: one makes it unique, the other decides where it belongs

Advanced · about 14 min

Goal: Separate RD from RT once and for all, then meet the nastiest trap in EVPN: get RT wrong and every session check passes, the route is plainly visible, and traffic still black-holes.

RD: letting routes coexist inside BGP

Start with the problem RD solves. BGP's decision process picks one best path among the paths for the same NLRI, and the rest are backup at best or not installed at all. Two situations in EVPN make sameness of NLRI dangerous: different tenants can absolutely reuse MACs and IPs (two customers both using 10.0.0.1, two cloned VMs with the same MAC), and during a VM migration leaf-1 and leaf-3 advertise the same MAC at the same time.

The RD (Route Distinguisher, 8 bytes) prefixes every route so that BGP sees distinct NLRIs. In production it is almost always auto-derived as my Router-ID plus the EVI id, which gives every EVI on every leaf a globally unique RD. Leaf-1's advertisement of MAC-X and leaf-3's advertisement of the very same MAC-X differ in RD, so they are two routes that can sit in the table together.

Two concrete benefits fall out of that. A route reflector can keep all the paths instead of just the best one, so during troubleshooting you can see that a MAC is currently claimed by two leaves. And MAC mobility can present itself as two paths with different sequence numbers, which is what gives lesson 6's arbitration something to compare. ★ Just remember that RD is not policy -- it provides uniqueness and nothing else, and it never decides which route gets imported where.

RT: deciding who imports it, and where

RT (Route Target) is an extended community, and it does exactly one job: decide which EVI (the L2 MAC-VRF) or which IP-VRF a route gets imported into. Every EVI has two lists: the export RT is the stamp I put on the routes I originate, and the import RT is the set of stamps I am willing to accept. On receiving a route the leaf compares them: if any RT the route carries appears in this EVI's import list, import it; if none does, leave it sitting in the BGP table untouched.

Auto-derived RTs are usually the ASN plus the VNI, and there is a trap hiding in that. It only lines up naturally when the whole fabric shares one AS (an iBGP-plus-route-reflector design). If you build eBGP per leaf per RFC 7938, with a different private ASN on every leaf, then every leaf derives a different RT and nobody imports anybody. So eBGP fabrics either configure RTs explicitly or turn on the vendor knob that ignores or rewrites the AS part of the RT. This is one of the most common day-one failures on a new fabric.

RT is also the only tool you have for multi-tenancy and route leaking: to let two VRFs talk, import each other's RT; to build hub-and-spoke, have the spokes import only the hub's export RT. ★ Getting the division of labour straight is enough: RD tells BGP that these are different routes; RT tells the receiving leaf where the route belongs -- and only RT can black-hole you silently.

★★★ The trap: RT is wrong and every check is green

The preset loaded here is this: the fabric was expanded with a new leaf-3, and VNI 10010 uses RT 65000:10 on every other leaf (the convention is to number by EVI), while leaf-3 was configured with 65000:10010 -- somebody used the VNI number as the EVI number. Nothing complains. The syntax is perfectly legal.

So you run your usual checklist and every single item is green: the underlay pings; show bgp l2vpn evpn summary has the session Established with counters incrementing; show bgp l2vpn evpn shows the remote MAC plainly sitting right there in the table; the VNI is up; the VTEP is up. And the hosts cannot talk.

The gap is between being in the table and being installed. The route really was received and really is stored in BGP's EVPN table (the Adj-RIB-In / global EVPN table), but it only reaches the MAC-VRF by passing the RT check, and only an imported route becomes a forwarding entry. An RT-mismatched route's status is exactly: visible, not imported. The failure is usually bidirectional too -- leaf-3's own advertisements are equally visible on the peers and equally not imported there.

★ The most deceptive part is its scope: typically only one VNI is broken while every other VNI on the same BGP session is perfectly fine. So everyone's attention gets pulled toward this one host, this one hypervisor, this one application, because the network is obviously healthy. One VNI broken plus healthy sessions is, almost always, RT -- or an L2VNI missing on one side.

How to actually find it

Ask the wrong question and you will never find this. Do not ask whether the session is up, ask whether the route was imported. In command terms: do not stop at show bgp l2vpn evpn, which only proves the route arrived. Go look at that route's RT extended communities and compare them item by item against the target EVI's import RT list.

On the '③ EVPN route table' page every route is labelled imported / ignored (RT mismatch) with the reason, and the RTs the route carries are shown side by side with the local import list. Find that MAC and look at its status on leaf-3, then go to '④ Forwarding' and send a packet to see which hop fails for lack of a forwarding entry.

After you fix it, compare the two tables again. ★ What should stay with you from this lesson is not a command but a shape: a route has two stages in its life -- advertised and received, then imported. Most EVPN failures of the everything-looks-fine-but-it-still-does-not-work kind happen in the second stage.

Key takeaways

  • RD provides uniqueness only (so two leaves can advertise the same MAC and coexist) and never decides anything about import
  • RT decides which EVI or VRF a route is imported into; a mismatched route is plainly visible in the BGP table and never reaches forwarding
  • Session Established, route visible, exactly one VNI broken -- suspect RT first, and watch out for auto-derived RTs on an eBGP-per-leaf fabric

Lesson 4 - ARP suppression and the distributed anycast gateway

Advanced · about 12 min

Goal: Get both halves straight: the gateway is always local, and ARP stops flooding -- both are things the EVPN control plane hands you for free.

What a centralized gateway costs you: hairpinning

In a traditional design the SVI (the gateway) lives on just one pair of boxes -- the aggregation switches, or the fabric's border leaves. So two hosts in different subnets under the same leaf reach each other by going leaf-1 -> border -> leaf-1, and cross-leaf inter-subnet traffic goes leaf-1 -> border -> leaf-3. The detour has a name: hairpinning, also called traffic tromboning.

It costs you on three fronts. Latency: what should be one hop becomes three. Capacity: all of the fabric's east-west L3 traffic is concentrated on that one pair of boxes, which undoes exactly the horizontal scaling the spine-leaf design was built to get. Reliability: that pair becomes a shared failure domain for every subnet of every tenant.

There is a subtler cost too. What a host keeps in its ARP cache is the gateway's MAC. When the gateway only exists on one specific pair of boxes, a VM that migrates to another rack still has to reach that MAC by flooding across the whole fabric -- your L3 gateway has become a dependency on a large L2 domain, which is the opposite of why you built a fabric.

The distributed anycast gateway: every leaf is the same gateway

The distributed anycast gateway is blunt about it: every leaf configures the exact same SVI IP for the subnet (say 10.10.10.1/24) and the exact same virtual MAC. From the host's point of view its default gateway is always one hop away on the local leaf. From the fabric's point of view inter-subnet traffic is routed at the ingress leaf, straight into the L3VNI and over to the egress leaf, with no detour anywhere.

Why does the MAC have to be identical, and not just the IP? Because what the host caches is the MAC. ★ After a live migration to another rack, the VM does not re-ARP -- it keeps sending frames to the gateway MAC in its cache, and because every leaf answers as the gateway on that MAC, the leaf in the new rack simply routes them: zero packet loss, and no gratuitous ARP required. Share the IP but not the MAC and it breaks the moment that cache entry is reused.

The price is that this gateway MAC is deliberately not unique in the fabric -- the same MAC exists on dozens of leaves. So it must not be advertised around as an ordinary RT-2, or leaves would end up pointing at each other. EVPN handles it specially (on the box this is the anycast-gateway, or sticky/static MAC, configuration). The thing for a learner to hold onto: the anycast gateway works only because the control plane knows this MAC is intentionally duplicated rather than in conflict.

ARP suppression: answer from the EVPN cache instead of flooding

ARP suppression rides the RT-2 from lesson 2: an RT-2 can carry a MAC and an IP together, which means the ARP binding a leaf learned locally is advertised along with the MAC. The result is that every leaf holds an IP -> MAC table for the entire VNI without ever having seen those ARP packets.

So when a local host ARPs for a remote host, the leaf answers locally from that cache and the broadcast never enters the fabric. In a large L2 domain ARP and ND are the single biggest source of BUM traffic, so suppressing them removes most of the cost that made flood-and-learn painful in lesson 1. This is the most concrete example of EVPN demoting flooding from the normal case to the exception.

Suppression is not free, and two boundaries matter. A cache miss still floods -- a host that has never spoken, or an entry that has aged out. And if you also disabled unknown-unicast flooding, that traffic is dropped rather than flooded. The classic victim is the silent host from lesson 1: a cluster VIP that only ever answers, or certain security appliances. When people say be careful with ARP suppression, this is the specific thing they mean.

Go to '④ Forwarding' and do two things. Toggle ARP suppression back and forth and watch the same ARP request switch between answered locally and replicated to N-1 VTEPs. Then walk an inter-subnet packet and confirm it is routed at the ingress leaf, with no hop detouring to a border.

Key takeaways

  • A centralized gateway causes hairpinning: all east-west inter-subnet traffic detours through one pair of border boxes, undoing horizontal scaling
  • A distributed anycast gateway gives every leaf the same SVI IP and the same virtual MAC, so a migrated VM never re-ARPs and never loses a packet
  • ARP suppression answers locally from the MAC+IP binding in RT-2, but a cache miss still floods and silent hosts are the classic victims

Lesson 5 - L2VNI vs L3VNI, and symmetric vs asymmetric IRB

Advanced · about 12 min

Goal: Separate what each kind of VNI is for, then see why symmetric IRB became the default -- the egress leaf needs no ARP state for remote subnets.

Two kinds of VNI, two jobs

The VNI is a single 24-bit field in the VXLAN header, but EVPN uses it to express two completely different things, and the configuration normally names them separately:

- L2VNI = one broadcast domain (a MAC-VRF, a bridge domain). Traffic inside a subnet uses it: the encapsulation keeps the original Ethernet header, the VNI says which broadcast domain, and the egress leaf just looks up its own MAC table and bridges.
- L3VNI = one routing domain (a tenant's IP-VRF). Only inter-subnet traffic uses it: the ingress leaf rewrites the Ethernet header (destination MAC becomes the egress leaf's Router MAC), sets the VNI to the L3VNI, and the egress leaf then does an IP lookup in that VRF.

★ The mnemonic: L2VNI answers which broadcast domain, L3VNI answers which routing table. The numbering follows from that and is easy to remember: one L2VNI per subnet, one L3VNI per tenant -- so a tenant with several hundred subnets normally has several hundred L2VNIs and exactly one L3VNI.

Asymmetric IRB: state grows with the subnet count

Historically there were two ways to do inter-subnet forwarding. Asymmetric IRB is route-and-bridge: the ingress leaf routes the packet locally out of the source subnet and into the destination subnet, encapsulates it with the destination subnet's L2VNI, and the egress leaf does nothing clever at all -- it just bridges.

That sounds convenient, and the bill arrives as state. To route locally into the destination subnet, the ingress leaf needs that subnet's SVI and it needs the destination host's ARP/ND entry. And any leaf in the fabric might be the ingress for some packet, so the conclusion is that every leaf has to hold the SVI and ARP state for every subnet -- even subnets it hosts not a single host of. State grows as subnets times hosts, which hits a ceiling fast on a large multi-tenant fabric.

It also makes troubleshooting harder: A -> B and B -> A travel in different VNIs (each direction uses the destination subnet's L2VNI), so the path is asymmetric and you have two sets of captures and counters to reconcile.

Symmetric IRB: why it became the default

Symmetric IRB is route-bridge-route: the ingress leaf does an IP lookup in the tenant's IP-VRF and encapsulates with the L3VNI, a pure transit VNI; the egress leaf decapsulates, does its own IP lookup in the same VRF, and then bridges into its local L2VNI to hand the packet to the host. Both directions use the same L3VNI, hence symmetric.

★ Here is why it won: the egress leaf only has to hold state for the subnets it actually hosts. A leaf never needs an SVI for a remote subnet and never needs a remote host's ARP entry -- it just needs the route in the VRF (from an RT-2 carrying an IP, or from an RT-5) plus the next-hop VTEP. That decouples per-leaf state from the total number of subnets in the tenant, and that decoupling is the whole scaling story.

Symmetric IRB does add consistency requirements. Every leaf in a tenant must use the same L3VNI number, must have that tenant's IP-VRF, and needs a Router MAC, which is carried in the RT-2's Router MAC extended community and is what the ingress leaf uses to rewrite the Ethernet header. Miss or mistype any one of those and the symptom is remarkably consistent: intra-subnet works perfectly, inter-subnet does not work at all. Lesson 3's question -- was the route imported -- applies here too, except you are looking at the IP-VRF rather than the MAC-VRF.

Go to '④ Forwarding' and walk an inter-subnet packet, watching three things: which table the ingress leaf looks up, whether the VNI in the encapsulation is an L2VNI or the L3VNI, and what the egress leaf looks up a second time.

Key takeaways

  • L2VNI answers which broadcast domain, L3VNI answers which VRF routing table; one L2VNI per subnet, one L3VNI per tenant
  • Asymmetric IRB requires every leaf to hold the SVI and ARP state for every destination subnet, so state grows with the total subnet count
  • Symmetric IRB transits over the L3VNI with a route lookup at each end, so the egress leaf only needs state for its local subnets -- which is why it became the default

Lesson 6 - Troubleshooting multi-homing (ESI, DF, split-horizon, aliasing) and MAC mobility

Troubleshooting · about 16 min

Goal: What each of multi-homing's four mechanisms repairs, then the trap that only ever surfaces at the application layer: no DF (or two of them) delivers duplicate BUM to the host.

ESI: dual-homing without a peer-link

To survive a single leaf failing, a server bonds two NICs and cables them to leaf-1 and leaf-2. From the server's side this is an ordinary LAG. The problem is on the network side: how do leaf-1 and leaf-2 know they are attached to the same host rather than to two different hosts?

The traditional answer is a vendor-proprietary MLAG or stack: run a peer-link between the two boxes, synchronize state across it, and pretend to be one device. EVPN's answer is completely different -- both leaves configure the same 10-byte ESI (Ethernet Segment Identifier) on that bundle, then each advertises I am attached to ESI X in an RT-4, and that is how they discover each other as neighbors on the same segment. No peer-link, no vendor pairing, and no limit of two -- the same ESI can have three or four leaves on it.

Multi-homing comes in two modes: single-active, where only one leaf forwards at a time and the other is pure standby, and all-active, where both forward and the server's LAG hashing spreads traffic across them. All-active is what everybody actually wants -- but the moment you allow one host to be reachable from two places at once, the next three mechanisms stop being optional.

DF and split-horizon: duplication and reflection first

DF (Designated Forwarder) fixes duplication. When BUM traffic floods in from the fabric, both leaf-1 and leaf-2 are attached to this Ethernet segment, and if both deliver it the host gets two copies. So the leaves use the candidate list they built from RT-4 to elect exactly one of them, per ES and per VNI, to deliver BUM onto that segment (the classic algorithm orders candidates by originating IP and picks one using the VNI modulo the candidate count). Note the per-VNI granularity: different VNIs can elect different leaves, which is itself a form of load sharing.

Split-horizon fixes reflection. A broadcast from the host enters through leaf-1 and gets flooded across the fabric, so leaf-2 receives a copy -- and leaf-2 is attached to that same segment, so it must never send that copy back down to the host. The mechanism is to tag the traffic with its source Ethernet segment (the ESI label carried in the per-ES RT-1; VXLAN deployments often implement it as local bias instead), and leaf-2 drops anything that came from the segment it is itself attached to.

Aliasing: then load sharing

Aliasing fixes load sharing. The host's MAC is only ever learned on one of the two leaves, whichever one the server's hashing sent that traffic to, so only that leaf originates an RT-2. If remote leaves forwarded strictly by RT-2, they would send 100% of the traffic to that single leaf and half the LAG you paid for would idle. The per-EVI RT-1 exists to close that gap: it tells remote leaves that this ESI is reachable through me too, so they can ECMP across both.

★ Grouped this way the three are easy to remember: all of them merely repair consequences of one fact, that the MAC really is reachable from two places. DF repairs delivered twice, split-horizon repairs sent back where it came from, aliasing repairs only ever sent to one side.

★★★ The trap: no DF, or two DFs

The fault in the preset loaded here is that RT-4 is not actually being exchanged between leaf-1 and leaf-2 -- maybe the ES-import RT does not match (a variant of lesson 3's trap), maybe one digit of the ESI was typed wrong so the two leaves do not recognize each other as being on the same segment, maybe a policy filtered the ES route out. The result is that each leaf concludes it owns the segment alone, so each elects itself DF. Now every BUM frame delivered to that segment is delivered twice.

What makes this hard to find is that unicast is completely fine. Pings succeed, most of the application works, the monitoring is green. Every symptom that does surface is at the application layer and looks nothing like a network problem: DHCP clients receiving two offers, duplicated ARP replies, multicast and IGMP receivers seeing doubled streams, cluster heartbeats receiving their own packets and calling a split brain, and the server's bonding driver reporting those baffling received-own-frame counters. Nobody thinks to look at DF election.

Why this fault is so hard to find

There is a mirror-image mistake worth learning at the same time: the two leaves configure the same ESI but on the wrong ports (the ESI got copy-pasted onto a different server's bundle). Now a leaf is DF for a segment it is not actually attached to, and BUM toward the real host is black-holed. Same class of misconfiguration, opposite symptom: one gives you duplicates, the other gives you nothing.

★ Keep this rule: duplicate frames on a multi-homed host means look at ESI, RT-4 and DF first, not at the host. Go to the '⑤ Multi-homing' page and read the DF election result -- healthy means exactly one DF per ES per VNI, and the page flags both-are-DF directly so you can line it up against how RT-4 converged.

MAC mobility: sequence arbitration and duplicate-MAC detection

The last piece is MAC mobility. When a VM live-migrates from leaf-1 to leaf-3, leaf-3 learns it locally and advertises an RT-2 for the same MAC carrying a MAC Mobility extended community whose sequence number is one higher than the value it had seen. The rule is simple: the higher sequence number wins. Leaf-1 receives a better route for a MAC it believed was local, stops advertising, withdraws its own route, and flushes the local entry.

★ This is precisely the hole in flood-and-learn from lesson 1: a move is signalled explicitly and converges in one BGP update, instead of waiting for a MAC aging timer to expire while traffic keeps going to the old leaf. The same event goes from minutes of black-holing to sub-second.

The sequence number has a second job: duplicate MAC detection. If the sequence number for one MAC keeps incrementing over a short window (a common default is 5 moves in 180 seconds), that is not a migration -- the same MAC genuinely exists in two places, or a loop is shuttling the frames back and forth. The leaf declares the MAC duplicate and freezes it, stopping the advertisement, rather than letting the whole fabric's tables oscillate. The trade-off is worth noticing: better to park one MAC than to let one broken host shake the control plane.

Root causes come in three flavours: a cloned VM whose MAC was never changed, two access ports cabled back to back into an L2 loop, and an ESI misconfiguration that makes one host look like two to the control plane. Go to '⑤ Multi-homing', trigger a migration first and watch the sequence number increment and the old leaf withdraw, then trigger a repeated flap and see on which move it gets declared duplicate.

Key takeaways

  • The ESI is the shared identifier saying these leaves attach to the same Ethernet segment -- no peer-link needed, and not limited to two leaves
  • All-active multi-homing needs all three: DF (BUM delivered once), split-horizon (never reflected back to the source segment), aliasing (both leaves receive traffic)
  • MAC mobility is arbitrated by sequence number; a sequence that keeps climbing in a short window means a duplicate MAC or a loop, not a migration