Cumulus RoCEv2 Configuration Guide

Introduction to Lossless RoCEv2

RDMA over Converged Ethernet (RoCE) allows high-throughput, low-latency data transfers between compute nodes at the memory-to-memory level, bypassing the CPU and OS network stack. While the original RoCE (v1) was restricted to a single Layer 2 broadcast domain, RoCEv2 enables routing across Layer 3 environments.

RoCEv2 achieves routability by encapsulating the InfiniBand (IB) transport protocol within standard Ethernet, IP, and UDP headers. A specific UDP destination port (4791) is used to identify RoCEv2 traffic, allowing standard IP routers to handle the packets.

Comparison of RoCE and RoCEv2 Frame Formats

The Three Pillars of RoCEv2 Flow Control

RoCEv2 employs a "Data Center Quantized Congestion Notification" (DCQCN) triad to maintain a lossless environment:

• DSCP (Traffic Classification): Marks IP headers with a QoS label to ensure packets are mapped to the correct traffic class across all switches.

• ECN (Congestion Signaling): A proactive mechanism that signals endpoints to "slow down" transmission before buffers overflow.

• PFC (Buffer Protection): A reactive "safety net" that pauses specific traffic classes on a hop-by-hop basis to prevent buffer exhaustion.

Congestion Management: AFD vs. WRED

• Weighted Random Early Detection (WRED): Marks traffic with ECN bits based on general buffer occupancy. It treats all traffic in a queue equally.

• Approximate Fair Drop (AFD): A more granular mechanism that distinguishes between "Elephant" (high-bandwidth) and "Mice" (low-bandwidth) flows. AFD is used to trigger ECN marking specifically on elephant flows, allowing short-lived "mice" communications to complete without being penalized by large data transfers.

Detailed Mechanism: Priority Flow Control (PFC)

PFC enables lossless Ethernet by managing congestion on a per-priority basis. Unlike standard Ethernet PAUSE frames, which halt all traffic on a link, PFC pauses only the specific Class of Service (CoS) assigned to RDMA (typically Priority 3), allowing management or heartbeat traffic to continue.

Threshold Logic: xOFF and xON

• xOFF: When a switch buffer reaches this high-water mark, it sends a PFC pause frame upstream.

• Buffer Headroom: This is the dedicated portion of the buffer that absorbs packets already "in flight" between when the pause is signaled and when the upstream sender halts. It is essentially a buffer for packet flight time over the wire.

• xON: Once the buffer drains below this lower threshold, the switch signals the upstream device to resume transmission.

PFC Storms & Watchdog: A "PFC Storm" occurs when a misbehaving host continuously transmits PFC frames, potentially deadlocking the fabric. The PFC Watchdog Interval monitors queues; if a queue is not drained within a specified timeout, the watchdog detects the deadlock and drops the undrained packets to restore network flow.

Detailed Mechanism: Explicit Congestion Notification (ECN)

ECN allows switches to signal congestion to end hosts without dropping packets by using the two least significant bits of the IP Header Type of Service (TOS) field.

ECN Bit Values

The ECN Interaction Loop

1. Marking (Switch): When buffer occupancy exceeds the WRED/AFD minimum threshold, the switch marks packets with 0x11 (CE).

2. Notification (Receiver NIC): The Receiver detects the 0x11 mark and generates a Congestion Notification Packet (CNP).

3. Throttling (Sender NIC): The Sender receives the CNP and reduces its transmission rate for that flow.

Contrast: ECN is a proactive, end-to-end mechanism, whereas PFC is a reactive, hop-by-hop link-level mechanism.

Step-by-Step RoCEv2 Packet Journey (DCQCN)

Lifecycle of an RDMA write operation across the fabric:

1. Classification (Sender NIC): The application initiates a write; the NIC sets the IP packet's DSCP value (e.g., 26).

2. Priority Mapping (Switch): The switch receives the packet and maps the DSCP to a Lossless Queue (e.g., Priority 3).

3. Buffer Monitoring (Switch Queue): The switch monitors buffer occupancy against WRED/AFD thresholds.

4. ECN Marking (Switch): If the AFD threshold for an elephant flow is reached, the switch sets the packet ECN bits to 0x11.

5. PFC Trigger (Switch/Sender): If congestion hits the xOFF threshold, the switch sends a PFC frame upstream to pause Priority 3 traffic.

6. Congestion Detection (Receiver NIC): The receiver detects the ECN 0x11 mark in the arrival packet.

7. CNP Generation (Receiver NIC): The receiver generates a CNP and sends it back to the sender.

8. Rate Reduction (Sender NIC): The sender receives the CNP and throttles its transmission rate, allowing fabric buffers to drain.

NVIDIA Cumulus Linux NVUE Configuration

• Cumulus Linux 5.11 RoCE Documentation

• Cumulus Linux 5.16 RoCE Documentation

Configuration on NVIDIA Spectrum ASICs is managed via the NVUE CLI.

Enable Lossless RoCE (Default Mode)

Command:

nv set qos roce
nv config apply -y

NVUE defaults to roce mode lossless. The commands nv set qos roce and nv set qos roce mode lossless are equivalent.

Verify:

nv show qos roce

Expected Result (key fields):

mode          : lossless
enabled       : on
traffic-class : 3
pfc           : enabled
ecn           : enabled

Lossless Single Ingress Pool

Command:

nv set qos roce mode lossless-single-ipool
nv config apply -y

Combines lossy/lossless pools for better absorption during burst traffic.

v5.16: Lossy Multi-TC with Packet Trimming

Command:

nv set qos roce mode lossy-multi-tc
nv set qos remark dscp-on-trim port-level
nv config apply -y

Specific Remarking Values (Standardized for AI POD):

• DSCP 21: Applied to host-downlinks (lossy-multi-tc-host-group).

• DSCP 11: Applied to network uplinks (lossy-multi-tc-network-group).

Verify DSCP Remarking:

nv show qos remark

Expected Result:

lossy-multi-tc-host-group    : dscp 21
lossy-multi-tc-network-group : dscp 11
dscp-on-trim                 : port-level

Technical Note: Spectrum-4 ASIC Reporting Engineers may notice a reporting discrepancy on Spectrum-4 switches. nv show qos roce may report an ECN threshold of 146.48 KB, while interface-level commands report 156 KB (159,744 bytes). The latter is the actual hardware-programmed value; this is expected behavior and does not indicate a configuration error.

Switch Verification Checklist

Example - Interface Counters:

nv show interface swp1 qos roce counters

Expected Result (healthy fabric):

pfc_rx_prio3    : 0
pfc_tx_prio3    : 0
ecn_marked      : 0
cnp_rx          : 0
cnp_tx          : 0

Operational Summary Table

NFS over RDMA (NFSoRDMA) on VAST

NFS over RDMA (also called RPC-over-RDMA) layers the NFS protocol on top of the RDMA transport instead of TCP. On VAST, this is the primary storage access method for AI/HPC workloads that require low latency and high throughput over a lossless RoCEv2 fabric.

How it works:

1. Client sends an initial MOUNT request over TCP to discover the NFS service.

2. Client then establishes an RDMA connection (RPC-over-RDMA) for data path I/O.

3. Data transfers bypass the CPU/kernel network stack via zero-copy RDMA operations.

Prerequisite: The lossless RoCEv2 fabric (PFC + ECN on Priority 3) described in the sections above must be configured end-to-end across client NICs, switches, and VAST CNodes before NFSoRDMA will perform reliably.

NFSoRDMA vs. NFS over TCP

Prerequisites

Load rpcrdma Module

Command:

modprobe rpcrdma
lsmod | grep rpcrdma

Expected Result:

rpcrdma               131072  0
rdma_cm                36864  3 rpcrdma,ib_iser,rdma_ucm

VAST NFS RDMA Mount (Example)

Command:

sudo mkdir -p /mnt/vast
sudo mount -t nfs -o \
  rw,noatime,nodiratime,vers=3,\
  rsize=1048576,wsize=1048576,namlen=255,\
  acregmin=60,acdirmin=60,soft,noacl,\
  forcerdirplus,proto=rdma,port=20049,nconnect=4,timeo=600,\
  remoteports=10.100.100.1-10.100.100.100,\
  spread_reads,spread_writes \
  10.100.100.1:/nfs3 /mnt/vast

Key VAST mount option for RDMA:

Verify NFSoRDMA Mount

Command:

findmnt /mnt/vast
mount | grep vast

Expected Result:

TARGET     SOURCE                FSTYPE OPTIONS
/mnt/vast  10.100.100.1:/nfs3    nfs4   rw,noatime,...,proto=rdma,nconnect=4,...

10.100.100.1:/nfs3 on /mnt/vast type nfs (rw,...,proto=rdma,nconnect=4,...)

Pass criteria: proto=rdma appears in mount options.

Basic NFSoRDMA I/O Test

Command:

ls -la /mnt/vast/
touch /mnt/vast/qa-test-$$ && rm /mnt/vast/qa-test-$$
dd if=/dev/zero of=/mnt/vast/qa-write-test bs=1M count=100 oflag=direct
rm /mnt/vast/qa-write-test

Expected Result:

(total) ...
100+0 records in
100+0 records out
104857600 bytes (105 MB, 100 MiB) copied, 0.12 s, 873 MB/s

Verify RDMA Transport is Active

Command:

rdma link
rdma statistic show | egrep "np_cnp_sent|rp_cnp_handled|roce_adp_retrans"

Expected Result (healthy):

link mlx5_0/1 state ACTIVE physical_state LINK_UP netdev enp225s0f0
    rate 200 Gb/sec

np_cnp_sent:        0
rp_cnp_handled:     0
roce_adp_retrans:   0

Client NIC QoS (Required for RDMA)

Command:

sudo mlnx_qos -i enp225s0f0

Expected Result:

Priority trust state: dscp
PFC configuration:
    priority    0   1   2   3   4   5   6   7
    enabled     0   0   0   1   0   0   0   0

PFC must be enabled on Priority 3 (Traffic Class 3). Priority trust state: pcp indicates a configuration mismatch.

NFSoRDMA Troubleshooting

Enable RPC debug (triage):

rpcdebug -m nfs -s all
rpcdebug -m rpc -s all
echo -n 'module rpcrdma +pflmt' > /sys/kernel/debug/dynamic_debug/control

Collect logs:

dmesg -T | grep -iE "nfs|rpc|rdma" > nfsordma_debug.out

Disable debug after collection:

rpcdebug -m nfs -c all
rpcdebug -m rpc -c all
echo -n 'module rpcrdma -pflmt' > /sys/kernel/debug/dynamic_debug/control

RoCEv2 Diagnostic and Troubleshooting Cheat Sheet on Server

Use this checklist to verify the RoCEv2 environment on ConnectX-based Linux systems.

Mellanox RoCEv2 Health Checklist

Example: NIC Detection

Command:

lspci | grep Mellanox

Expected Result:

e1:00.0 Ethernet controller: Mellanox Technologies MT2910 Family [ConnectX-7]
e1:00.1 Ethernet controller: Mellanox Technologies MT2910 Family [ConnectX-7]

Example: RDMA Devices

Command:

ibv_devices

Expected Result:

    device          	   node GUID
    ------          	----------------
    mlx5_0          	e46dab03005db966
    mlx5_1          	e46dab03005db967

Example: RoCE Version (GID Type)

Command:

show_gids

Expected Result:

DEV	PORT	INDEX	GID					IPv4  		VER	DEV
---	----	-----	---					------------  	---	---
mlx5_0	1	0	fe80:0000:0000:0000:e66d:abff:fe5d:b966			v1	enp225s0f0
mlx5_0	1	1	fe80:0000:0000:0000:e66d:abff:fe5d:b966			v2	enp225s0f0
mlx5_0	1	10	0000:0000:0000:0000:0000:ffff:ac15:5214	172.21.82.20  	v2	enp225s0f0

GID Type 2 (v2) confirms RoCEv2 is active.

Example: Device-to-Interface Mapping

Command:

ibdev2netdev

Expected Result:

mlx5_0 port 1 ==> enp225s0f0 (Up)
mlx5_1 port 1 ==> enp225s0f1 (Up)

Performance Verification and Statistics

Validate throughput and monitor hardware counters to distinguish between a healthy fabric and one experiencing "PFC Storms" or buffer exhaustion.

Critical Congestion Counters

Command:

rdma statistic show

Expected Result (healthy fabric):

np_cnp_sent:        0
rp_cnp_handled:     0
roce_adp_retrans:   0
roce_slow_restart:  0

Continuous increases in np_cnp_sent indicate active congestion. Increasing roce_adp_retrans suggests the lossless path is failing.

Bandwidth Test

Command (server):

ib_write_bw -d mlx5_0 -a

Command (client):

ib_write_bw -d mlx5_0 -a <server_ip>

Expected Result (200G link):

************************************
* Waiting for client to connect... *
************************************
---------------------------------------------------------------------------------------
                    RDMA_Write BW Test
 Dual-port       : OFF          Device         : mlx5_0
 Number of qps   : 1            Transport type : IB
 Connection type : RC           Using SRQ      : OFF
 Link type       : Ethernet
 GID index       : 6
---------------------------------------------------------------------------------------
 #bytes     #iterations    BW peak[MB/sec]    BW average[MB/sec]   MsgRate[Mpps]
 8388608    5000             22825.74            19233.22                  0.002404
---------------------------------------------------------------------------------------

PFC Counters on Host NIC

Command:

ethtool -S enp225s0f0 | egrep "pfc|pause|prio"

Expected Result (healthy fabric):

rx_prio3_pause: 0
rx_prio3_pause_duration: 0
tx_prio3_pause: 0
tx_prio3_pause_duration: 0

Quick One-Liner Health Check

Command:

echo "=== RDMA Devices ==="; ibv_devices; \
echo; echo "=== Mapping ==="; ibdev2netdev; \
echo; echo "=== Link Layer ==="; ibstat | grep "Link layer"; \
echo; echo "=== RoCE GIDs ==="; show_gids; \
echo; echo "=== Congestion Counters ==="; rdma statistic show | egrep "roce_adp_retrans|np_cnp_sent|rp_cnp_handled"; \
echo; echo "=== PFC Counters ==="; ethtool -S eth0 | egrep "pfc|pause|prio"

Expected Result (summary):

=== RDMA Devices ===
device    node GUID
mlx5_0    248a070300a0b1c2

=== Mapping ===
mlx5_0 port 1 ==> eth0 (Up)

=== Link Layer ===
        Link layer: Ethernet

=== RoCE GIDs ===
mlx5_0  1  2  ...  10.0.0.100  v2

=== Congestion Counters ===
np_cnp_sent: 0
rp_cnp_handled: 0
roce_adp_retrans: 0

=== PFC Counters ===
rx_prio3_pause: 0
tx_prio3_pause: 0