Lesson text for this page (click to expand)

InfiniBand

Lessons (6, study in order)

Lesson 1 · ★★★ Getting three words straight

Beginner · about 8 min

Goal: Get the relationship between RDMA / RoCE / InfiniBand right — get it wrong and everything after this is muddled.

★★★ The question "InfiniBand vs RDMA" does not hold

The conclusion first: InfiniBand *is* an RDMA fabric.

The two words are not at the same level:

· RDMA is a set of semantics and interfaces (verbs) — one-sided WRITE/READ, two-sided SEND/RECV, zero copy, kernel bypass, hardware offload. It specifies *how an application gets the NIC to move memory directly*.
· InfiniBand and Ethernet are two networks that carry it — they specify *what a packet looks like, where addresses come from, how switches forward, and how loss is avoided*.

★★ So the correct comparison is: the same RDMA, running on two different networks.

★★★ Which gives RDMA two mainstream ways to run

| Flavour | Network underneath | In one line |
|---|---|---|
| InfiniBand | native IB | Its own stack from link layer to transport; needs dedicated HCAs and IB switches |
| RoCEv2 | Ethernet | Carries the IB transport layer inside UDP/IP, reusing Ethernet kit |

★ There is also iWARP (RDMA over TCP), with a much smaller ecosystem; not covered here.

★★★ Remember this: the "RDMA" in RoCE and the "RDMA" in IB are the same thing — even the transport layer is the same (IB transport); only the wrapper differs.

★★ Which is why this module's simulation is three-way

To answer both "vs Ethernet" and "vs RDMA" at once, the simulation runs three paths:

① Plain Ethernet + TCP/IP
② Ethernet + RDMA (= RoCEv2)
③ InfiniBand + RDMA

★★★ And then you see a very clean pattern:
· ①↔②③ differ inside the host (copies, syscalls, peer CPU)
· ②↔③ are identical inside the host; all the difference is at the network layer

That pattern is itself the answer.

Key takeaways

  • ★★★ RDMA is semantics/an interface; InfiniBand and Ethernet are networks that carry it
  • ★★★ "IB vs RDMA" should be asked as "IB vs RoCE" — same verbs, different network
  • ★★ RoCEv2 = the IB transport layer carried inside UDP/IP

Lesson 2 · ★★★ vs plain Ethernet: who deserves the credit

Beginner · about 10 min

Goal: Where IB beats plain Ethernet — and why most of that credit does not belong to IB.

The difference splits into two layers

Layer one: inside the host. Plain TCP needs 2 copies, 2 syscalls, both kernel stacks, and the peer CPU dragged in by an interrupt throughout.
RDMA removes all of it: 0 copies, 0 syscalls, and with one-sided operations the peer CPU is not involved at all.

Layer two: the network layer. Who hands out addresses, who writes the switch forwarding table, whether losslessness is designed in or bolted on.

★★★ The key point: layer one's credit belongs to RDMA, not InfiniBand

This is the easiest thing to get wrong.

Look at the simulation page: ② RoCE runs on ordinary Ethernet, yet its "inside the host" numbers are identical to ③ InfiniBand's — copies both 0, syscalls both 0, CPU-using stages exactly the same.

★★★ In other words: to get the bulk of the benefit you do not need to switch to InfiniBand at all — enabling RoCE on the Ethernet you already have is enough.

★★ InfiniBand's own contribution is layer two — smaller, but decisive in specific situations.

★ Do not forget "the difference is not on the wire"

All three paths use the same links for the HCA → switch → HCA segment.

IB's only advantage there is slightly quicker switch forwarding (cut-through, order of ~0.15µs vs Ethernet ~0.6µs) and a shorter header.

⚠️ With no congestion that difference is small — do not treat it as IB's main selling point.

Key takeaways

  • ★★★ "IB is faster than Ethernet" needs splitting: mostly from using RDMA, a little from changing fabric
  • ★★★ To check this: ② RoCE's "inside the host" numbers are identical to ③'s
  • ★ All three paths are identical over the wire segment

Lesson 3 · ★★★ vs RoCE: all the difference is at the network layer

Intermediate · about 12 min

Goal: Same verbs, different network. Exactly which items the difference lands on.

★★★ Inside the host: identical everywhere

The same ibv_post_send(), the same QP/CQ/MR concepts, the same zero copy, the same kernel bypass, the same hardware offload.

★★ Even the costs are the same: memory still has to be registered, the QP still has to be pushed all the way to RTS, and a two-sided SEND still needs the peer to post a RECV first.

★★★ Switching to InfiniBand skips not one of RDMA's prerequisites — because IB *is* RDMA. Scenario ④ ("forgot to register memory") exists to demonstrate exactly this: ②③ fail together, only ① keeps sending.

★★★ At the network layer: different everywhere

This is what InfiniBand actually is. Four items:

| | RoCEv2 (Ethernet) | InfiniBand |
|---|---|---|
| Address | factory-burned MAC + configured IP | ★ LID assigned by the SM |
| Forwarding table | switch floods and self-learns | ★ computed centrally by the SM and written down |
| Losslessness | bolted-on PFC + ECN | ★ credit-based, written into the link-layer spec |
| Bring-up | has to borrow plain TCP | IB's own CM/SM |

★★ Lessons 4 and 5 cover the first two items and the third in detail.

★★ An easily missed one: RoCE bring-up still needs TCP

Before RDMA can send anything it must know the peer's QP number and address, and RDMA cannot carry that information itself (the QP does not exist yet).

· RoCE: usually borrows plain TCP — so the TCP/IP stack on a RoCE host is not optional
· IB: can use IB's own CM (connection management) + SM management path, with no TCP/IP stack

★ Conversely, IPoIB is common in IB clusters (emulating IP over IB) — that exists so legacy socket-only programs can run, not for RDMA's benefit.

Hands on: see the pattern

Go to the simulation page; the comparison table splits into "inside the host" and "network layer".
★★ The page marks rows where ②③ are the same in grey and where they differ in green — you will see the first group almost all grey and the second almost all green.

Key takeaways

  • ★★★ Inside the host ②③ are identical everywhere (same verbs, same costs too)
  • ★★★ At the network layer ②③ differ everywhere: address / forwarding table / losslessness / bring-up
  • ★★ RoCE bring-up borrows plain TCP; IB uses its own CM/SM

Lesson 4 · ★★★ Addresses are handed out: the subnet manager

Intermediate · about 10 min

Goal: IB's most counter-intuitive property: with no subnet manager an IB network has no addresses at all.

★★★ Ethernet: self-organising

· Address: the MAC is burned into the NIC at the factory, globally unique, and nobody manages it
· Finding neighbours: ARP simply shouts "who has this IP"
· Switches: flood on an unknown destination, and learn an entry from every source MAC seen

★★ The whole mechanism has no central role at all. Plug in and it works — one of the reasons Ethernet took over the world.

★★★ InfiniBand: centrally managed

· Address: a port address is a LID (16-bit local ID), assigned by the subnet manager (SM). Across subnets there is also a GID (128 bits, much like an IPv6 address).
· Switches: the forwarding table is an LFT (linear forwarding table), which the SM discovers the topology for, computes routes for, and writes into every switch.

★★★ So the consequence of no SM is hard:
ports stay in INIT, there is no LID, and switches do not know where to forward → not a single data packet can be sent.

★★ On Linux the SM is usually opensm (it can also run inside the switch). Production runs a primary and a standby SM, because it is a component that must stay alive.

⚠️ But the SM is not in the data path

Do not read this as "every packet has to ask the SM" — it does not.

The SM's work is one-time: at subnet bring-up (and on topology changes) it assigns LIDs, computes routes and writes forwarding tables. After that data packets never go through the SM; the HCA sends by LID and switches forward straight from their tables.

★ That is what the "one-time setup" tag on that step in the simulation means.

Hands on: turn the SM off

The current scenario is ② ("subnet manager not running"). Just run it:
★★★ you will see only the third diagram fail and go grey, while ①② (both Ethernet) carry on exactly as before.

Key takeaways

  • ★★★ An IB LID is assigned by the SM, not burned in at the factory
  • ★★★ The switch forwarding table (LFT) is also computed centrally by the SM and written down
  • ⚠️ The SM only works at bring-up / on change — data does not go through it
  • ★★ Production runs a primary and a standby SM — that is extra operational burden IB brings

Lesson 5 · ★★★ Losslessness: designed in vs bolted on

Intermediate · about 12 min

Goal: Both are called "lossless", but IB and Ethernet implement it a level apart.

Why RDMA is so afraid of loss

Classic RoCE hardware retransmission is go-back-N: lose one packet and everything already sent after it has to go again.

At 100Gbps there can be hundreds of packets in flight, so 1% loss can cut throughput to a fraction — a completely different concept from TCP's graceful degradation.

⚠️ To be fair: newer NICs support selective repeat (resending only the lost packet), which is much better than the go-back-N era.

★★★ Ethernet's answer: bolt on two mechanisms

· PFC (Priority Flow Control): when a switch buffer is nearly full it sends a "pause" frame upstream.
⚠️ This is a blunt instrument: an entire priority queue stops → head-of-line blocking; cascaded it becomes a PFC storm; with a loop, deadlock in the extreme.
· ECN / DCQCN: the switch marks packets, the receiver reflects the mark, and the sender reduces its rate.

★★★ The order of precedence matters: ECN rate reduction first, PFC only as a last resort.

★★ But the crucial part is that both have to be configured by a human; get it wrong and it degrades to ordinary lossy Ethernet.

★★★ IB's answer: written into the link-layer spec

IB uses credit-based flow control:

the receiver grants credit for as much buffer space as it has, and a sender holding no credit sends not one byte.

★★★ So queues simply never overflow, and "dropping because of congestion" does not exist.

★★ And this is not a feature that can be switched off — it is part of the IB link-layer specification, with no corresponding knob. That is the difference between "designed in" and "bolted on".

⚠️ To be fair: IB is not drop-free either (bit errors and failover still lose packets), and credit backpressure also spreads congestion upstream (victim flows), which is why IB has its own congestion control to tune.

Hands on: look at retention

The current scenario is ③ ("congestion + Ethernet losslessness off"). Run it and look at the throughput retention row:
① TCP drops a chunk, ② RoCE collapses, ③ IB barely moves.

★★ Then turn "Ethernet losslessness" back on and re-run — ② recovers a long way. That shows RoCE can be made to work, but it has to be configured.

Key takeaways

  • ★★★ IB: credit-based, written into the link-layer spec, cannot be turned off
  • ★★★ RoCE: PFC + ECN, bolted on, must be configured, can be misconfigured
  • ★★ PFC is a blunt instrument (head-of-line blocking / storms / deadlock); ECN rate reduction is the workhorse
  • ⚠️ The difference is whether the default is safe, not "one works and the other does not"

Lesson 6 · ★ Which should you pick (no hype either way)

Troubleshooting · about 8 min

Goal: When InfiniBand is genuinely worth it, and when RoCE is enough.

★ Ask the cheap question first

"Am I not even using RDMA yet?"

If so, the first step is not buying IB — it is enabling RoCE on the Ethernet you already have.
Lesson 2 already proved it: the bulk of the benefit (zero copy / kernel bypass / hardware offload) is available on Ethernet.

★★ When InfiniBand really is worth it

· Very large training / HPC: thousands of cards doing all-reduce, where congestion is the norm rather than the exception — here "losslessness by design" is worth a great deal
· Extreme tail-latency sensitivity: IB switches are cut-through and paths are controllable (the SM computed them), so the tail is steadier than self-learning Ethernet with PFC
· You already have an IB ecosystem: MPI, scheduling and monitoring all built around IB
· You need adaptive routing / fine topology control: the SM computing routes centrally is an advantage here

★★ When not to bother

· Only a few dozen machines and congestion is not the norm → RoCE is enough, and you save a whole set of hardware and operations
· You must route across subnets / sites → RoCEv2 is IP-based and naturally routable; IB across subnets needs a router and is far more awkward
· Nobody on the team knows opensm / ibdiagnet → one more component that must stay alive (with a standby), and a separate troubleshooting toolchain
· The same machines also carry lots of ordinary TCP traffic → one Ethernet fabric doing both jobs is cheaper

★ In one sentence

"Whether to use RDMA" is the big decision; "IB or RoCE" is the small one.

The former decides whether you can drop the copies and the CPU cost (a multiple-fold difference); the latter decides how steady you are under congestion, and how much money and operational effort it takes.

★★ Do the former first, then consider the latter.

Key takeaways

  • ★★★ Ask "am I using RDMA at all" before "should I switch to IB"
  • ★★ IB is worth it at large scale, where congestion is normal, for tail latency, or where the IB ecosystem already exists
  • ★★ Skip it when the scale is small, you need cross-subnet routing, or nobody can operate an SM
  • ★ "Whether to use RDMA" is the big decision, "IB or RoCE" the small one