Lessons (6, study in order)

Lesson 1 · First, count how long the ordinary path is

Beginner · about 8 min

Goal: List out every cost TCP pays to send one message — without that list you cannot feel what RDMA saves.

What actually happens on one write()

One write() from the application sends the data down a path this long:

① Syscall: trap into the kernel, a user/kernel transition
② Copy #1: from the user buffer into the kernel socket buffer (the CPU moves it by hand)
③ Kernel stack: segmentation, checksums, TCP/IP headers, route lookup (the CPU computing)
④ DMA to the NIC → onto the wire
⑤ The peer NIC receives it, DMAs it into the kernel buffer
⑥ Interrupt + softirq: the peer CPU is preempted and pulled in to work
⑦ The peer's kernel stack: verify, reorder, reassemble, ACK
⑧ Copy #2: kernel buffer → the peer's user buffer
⑨ The peer's read() returns — one more context switch

★ Count them: 2 copies, 2 syscalls, both kernel stacks, and the peer CPU busy throughout.

★ Which costs scale with the data volume

Copies and stack processing are per byte — the bigger the data the more they hurt.
Syscalls and interrupts are fixed costs — the smaller the message, the larger their share.

★★ Remember that distinction; lesson 6 uses it to judge whether RDMA is worth it.

Hands on: look at the TCP path first

Go to the traffic simulation page and run it, looking only at the TCP side first.
★ Note that both kernel boxes light up on the canvas, and look at the "stages that use CPU" figure.

Key takeaways

• ★ TCP sending one message: 2 copies + 2 syscalls + both kernel stacks
• ★ The peer CPU is dragged in throughout (interrupt + stack + copy)
• Copies and stack cost scale per byte; syscalls and interrupts are fixed costs

Lesson 2 · ★★★ The three shortcuts RDMA takes

Intermediate · about 12 min

Goal: Zero copy, kernel bypass, hardware offload — all three together is what RDMA is.

★★★ Shortcut one: zero copy

The NIC DMAs straight out of the application's own memory and sends it.

There is no "copy into the kernel first" step — copies go from 2 to 0.

★ Precondition: that memory must have been registered in advance (lesson 4).

★★★ Shortcut two: kernel bypass

Sending is not "call write() and trap into the kernel". It is post a WQE onto the QP from user space and ring a doorbell — the queue is memory mmap'd into user space.

Reaping completions (CQEs) is user-space polling too, so syscalls go from 2 to 0 and so do the context switches.

★★★ Shortcut three: hardware offload

Segmentation, sequence numbers, acks, retransmission and congestion control all run in the NIC chip.

TCP does all of that in kernel software — which is the CPU computing.

★★ So the peer CPU is not interrupted either: the data lands in memory and nobody is woken up.

★ One segment is exactly the same

NIC → switch → NIC is identical for RDMA and TCP, and takes the same time.

★★★ So the honest statement is: the difference is not on the wire. What RDMA saves is entirely the stages inside the machine.

Hands on: put the two paths side by side

Go to the traffic simulation page and run the comparison — the same parameters, TCP once and RDMA once, drawn one above the other.
★ Watch the two kernel boxes go grey with a "(bypassed)" label.

Key takeaways

• ★★★ Three shortcuts: zero copy + kernel bypass + hardware offload
• ★★ Copies 2→0, syscalls 2→0, and the transport layer moves from kernel software into NIC hardware
• ★★★ The difference is not on the wire — the cable segment is identical

Lesson 3 · ★★★ One-sided vs two-sided

Intermediate · about 10 min

Goal: Whether the peer's CPU takes part at all — the most counter-intuitive and most useful distinction in RDMA.

★★★ One-sided (WRITE / READ)

This side holds the peer's memory address plus an rkey and reads or writes that memory directly.

★★★ The peer's CPU is not involved at all — no interrupt, no read(), and the peer application does not know it happened.

⚠️ But be precise about it: "the CPU is not involved" is not "the data does not arrive". It lands in the peer application's memory, every byte of it — that memory is the destination. What is skipped is the peer's CPU, not the peer application.

★ So how does the peer find out? By separate arrangement: polling a flag in memory, or the sender following up with a small two-sided SEND. That is what makes one-sided harder to use.

★★ Two-sided (SEND / RECV)

The peer must post a RECV in advance so the data has a registered buffer to land in, and afterwards it reaps a CQE to learn the data arrived.

So its CPU is lightly involved — but note it still does no protocol work and no copying: still zero copy, still no kernel stack.

⚠️ If the RQ runs empty you get RNR (Receiver Not Ready). Standard practice: replenish one RECV for every one consumed.

★ What each is good for

· Two-sided: control messages, requests, anything where the peer needs to *know* something arrived
· One-sided: bulk data, and any case where you want to leave the peer CPU completely alone

★★ A common real-world combination: two-sided for small control messages, one-sided for bulk data — the control message tells the peer "the data is already in your memory".

Hands on: compare the peer-CPU row

Switch between WRITE (one-sided) and SEND (two-sided) and watch the "peer CPU involvement" row, plus whether the B·App box changes.

Key takeaways

• ★★★ One-sided: the peer CPU is not involved — but the data still lands in its memory
• ★★ Two-sided: post a RECV in advance, reap a CQE afterwards — lightly involved, still zero copy
• ★ Common combination: two-sided for control, one-sided for data

Lesson 4 · ★★ The cost: three prerequisites you cannot dodge

Intermediate · about 10 min

Goal: RDMA is not "install a NIC and it is fast" — these three have to be in place first.

★★ Prerequisite one: memory must be registered (Memory Region)

For the NIC to touch memory while bypassing the CPU, it has to know in advance which memory it may touch.

Registration makes the kernel pin the physical pages so they cannot be swapped out, build the address translation, and hand the NIC an lkey / rkey.

★ No registration, no key, no permission — not one byte gets sent.
★★ Registration is not cheap, so real programs register a large pool at startup and manage it themselves; putting registration on the hot path negates much of the benefit.

★★ Prerequisite two: the QP must be pushed to RTS

A QP has to walk its state machine: RESET → INIT → RTR → RTS, and only at RTS can it really send.

★ Stopping at INIT means you can post RECVs but cannot send or receive.

★★★ Prerequisite three: information has to be exchanged over an ordinary network first

Pushing to RTR/RTS needs the peer's QP number and GID, and RDMA cannot carry that itself — the QP does not exist yet.

★★★ So there is a counter-intuitive conclusion: RDMA's connection setup is done over plain TCP.

★ That is why every verbs example opens a TCP socket first to swap QP number / GID / rkey / target address.

Hands on: take a prerequisite away

The current scenario has memory registration off. Run it:
★★★ you will see RDMA fail at the very first step while TCP sends fine under the same conditions.
★ Try the "QP not in RTS" scenario as well.

Key takeaways

• ★★ Three prerequisites: register memory / push the QP to RTS / exchange information out of band
• ★★★ RDMA's connection setup relies on plain TCP — it cannot bootstrap itself
• ★ Registration is expensive: register a pool up front, keep it off the hot path

Lesson 5 · ★★★ Why RoCE must have a lossless network

Intermediate · about 12 min

Goal: The same loss costs RoCE far more than it costs TCP — and why.

TCP under loss: patch only what was lost

TCP has fast retransmit + SACK: it resends only the packets actually lost, then halves the congestion window and grows back.

★ Throughput drops a step, but it degrades gracefully rather than collapsing.

★★★ Classic RoCE under loss: go-back-N, everything after it goes again

Classic RoCEv2 hardware retransmission is go-back-N: lose one packet and everything already sent after it has to be resent.

★★★ At 100Gbps there can be hundreds of packets in flight, so 1% loss can cut throughput to a fraction.

⚠️ One fair addition: newer NICs support selective repeat (resending only the lost packet), which is far better than the go-back-N era — but the engineering conclusion that "RoCE needs near-lossless" still holds today.

★★★ Hence the network has to be made "lossless"

Two mechanisms, and the order between them matters:

· ECN / DCQCN leads: the switch marks packets, the receiver reflects the mark, and the sender reduces its rate proactively. Gentle, end-to-end.
· PFC is only a backstop: send a PAUSE frame when a buffer is nearly full.
⚠️ A blunt instrument — a whole priority queue stops → head-of-line blocking; cascaded it becomes a PFC storm; with a loop, deadlock in the extreme.

★★ Get the order backwards (leaning on PFC) and the whole fabric jams as soon as load rises.

Hands on: see the retention gap with your own eyes

The current scenario is 1% loss with losslessness off. Run it and look at the throughput retention row: TCP keeps about nine tenths, RoCE less than three.
★★ Then open "losslessness" in the experiment parameters and re-run — retention goes straight back to 100%.

Key takeaways

• ★★★ Classic RoCE is go-back-N: one loss resends the whole window after it
• ★★★ Hence a RoCE deployment must first make the network near-lossless
• ★★ ECN/DCQCN leads, PFC is only a backstop — get that backwards and the fabric jams
• ⚠️ What is claimed is retention, not "RoCE is necessarily slower than TCP"

Lesson 6 · ★ When to use it and when not to

Troubleshooting · about 8 min

Goal: Where RDMA pays off, and where it is not worth the trouble.

★ The three situations where it pays off most

· Distributed storage: reading and writing remote memory or SSDs, where the CPU should not be spending its time copying
· Multi-machine multi-GPU training: gradient exchange is huge and frequent, and with GPUDirect the NIC can DMA straight in and out of GPU memory
· In-datacentre high-throughput RPC / HPC: a controllable network where latency and CPU both matter

★★ When not to bother

· Across the public internet: no control over loss, and RoCE's sensitivity to loss makes it unusable
· Small volumes of scattered small messages: the fixed cost of registration and connection setup outweighs the gain
· Nobody can operate PFC/ECN: a misconfigured lossless network is worse than plain TCP
· The code cannot be changed: verbs is a different programming model, not a drop-in for sockets

★ Middle options — do not forget these exist

It is not a binary choice between "plain TCP" and "rewrite everything in verbs":

· Bigger MTU / more queues / interrupt affinity — cheap tuning, often surprisingly effective
· DPDK / user-space networking — kernel bypass without RDMA semantics
· NVMe-oF / libraries such as UCX — someone else has already wrapped the verbs for you

★★ Try the cheap options before the expensive one.

Hands on: see how message size changes the conclusion

The current scenario is a 1MB message. Compare it with 4KB:
★ small messages win on latency (fixed costs are a large fraction); large messages see the latency gap diluted but the CPU and memory bandwidth saved become more valuable.

Key takeaways

• ★ Worth it: distributed storage / multi-machine training / in-datacentre high-throughput RPC
• ★★ Not worth it: the public internet / scattered small messages / nobody to operate the lossless network
• ★ Remember the middle options: MTU and queue tuning, DPDK, or a ready-made library