RoCE v2 and InfiniBand employ fundamentally different congestion notification mechanisms that determine their behavior under the incast and persistent congestion patterns common in AI training fabrics. RoCE relies on ECN (Explicit Congestion Notification, RFC 3168) at the IP layer: when a switch port's queue depth exceeds a configurable threshold K_min (typically 60-80% of the per-port buffer), the switch marks the CE (Congestion Experienced) codepoint in the IP header of passing packets. The receiver copies the ECN mark into the corresponding TCP ACK (or RoCE's ACK-like CNP—Congestion Notification Packet), and the sender's DCQCN (Data Center Quantized Congestion Notification) algorithm responds by reducing its transmission rate. The DCQCN algorithm defines a rate decrease factor α = 0.5 × (1 - β) + β × R, where β is the probability of ECN marking in the current sampling window and R is the current rate. When an ECN-marked CNP arrives, the sender reduces rate by a factor of (1 - α/2) with a probability proportional to the marking ratio. The convergence time for DCQCN to stabilize after a congestion event is approximately 3-5 RTTs (300-500 μs for a 100 μs RTT), compared to InfiniBand's FECN/BECN mechanism that converges in 1-2 RTTs. This slower convergence makes RoCE more susceptible to incast congestion (all-reduce fan-in at the rail switch) where multiple GPUs simultaneously send data to a single receiver, creating a transient queue buildup that persists for 100-200 μs—within DCQCN's convergence window.

InfiniBand's FECN (Forward Explicit Congestion Notification) and BECN (Backward Explicit Congestion Notification) operate at the link layer within the InfiniBand packet header. When a switch port's buffer occupancy exceeds the threshold K_FECN (typically configurable between 0-100% of the per-VL buffer, default 75%), the switch sets the FECN bit in the packet's Local Route Header (LRH) as it forwards the packet toward the destination. The destination NIC, upon receiving a packet with FECN set, sends a congestion notification packet (CNP) to the source with the BECN bit set. Unlike RoCE's ECN scheme where the CNP is a full packet requiring buffer allocation and queuing, InfiniBand's CNP is an 8-byte Link Layer Congestion Notification (LLCN) frame that is inserted directly into the send queue bypassing the standard packet buffer—eliminating the 150-300 ns of queuing delay associated with RoCE CNP processing. The InfiniBand sender's adaptive rate control algorithm, defined in the IBTA specification as the Congestion Control Table (CCT), uses a rate decrease factor of τ = 0.75 per received BECN, with a recovery time constant of τ_recovery = 10-100 μs (typically 15 μs per BECN-free interval). This 0.75× reduction per BECN means that after receiving two consecutive BECN frames, the sender's rate drops to 0.75² = 0.56× of the original—a 44% reduction in one RTT (100 μs), compared to DCQCN's approximately 25% reduction over the same interval.

The reaction to transient vs. persistent congestion differs significantly between the two mechanisms, impacting training throughput stability. Under transient congestion (a single 200 μs queue buildup from an NCCL all-reduce burst), InfiniBand's BECN may trigger a rate reduction just as the congestion is clearing, causing the sender to operate at a sub-optimal rate for the recovery period (typically 5-10 RTTs, or 500-1,000 μs). RoCE's DCQCN employs a probabilistic marking approach where only a fraction of packets traversing the congested queue are ECN-marked—the marking probability P_mark = (Q_current - K_min) / (K_max - K_min) where Q_current is the instantaneous queue depth, K_min is the minimum threshold, and K_max is the maximum threshold (typically 2× K_min). This probabilistic marking means that if the congestion is transient (Q_current exceeds K_min for only 50 μs), only a fraction of packets during that window are marked, reducing the number of rate reduction events. InfiniBand's FECN marking is deterministic: when Q exceeds K_FECN, every packet traversing the port is marked until Q drops below K_FECN minus a hysteresis margin K_hyst (typically 5-10% of the per-port buffer). The deterministic marking causes a burst of BECNs that triggers multiple consecutive rate reductions, over-correcting for transient congestion. Our comparison model simulates both marking strategies under the all-reduce traffic pattern of a 4,000-GPU cluster, showing that RoCE's probabilistic ECN marking achieves 4% higher average throughput but 12% higher tail latency (99th percentile) compared to InfiniBand's deterministic FECN/BECN marking.

The credit-based flow control interaction with congestion notification is a critical difference that determines how each fabric recovers from congestion. InfiniBand's link-layer credit scheme ensures that once the sender is authorized (credited) to send a packet, that packet will never be dropped at the receiver due to buffer exhaustion—credit management is per-VL and independent of the congestion notification mechanism. This means that after a BECN-triggered rate reduction, the sender's in-flight packets are guaranteed delivery, and recovery is smooth: the sender gradually increases its rate using the CCT's table-based rate recovery (τ_recovery per BECN-free interval). RoCE's lack of credit-based flow control at the link layer means that during DCQCN's rate reduction, the sender may still be transmitting at the old rate for packets already in flight, and if the switch buffer overflows before rate reduction takes effect, tail-drop packet loss occurs. RoCE RC (Reliable Connection) transport interprets this loss as a missing packet and triggers retransmission, adding an extra 2× RTT (200 μs) for the retransmission plus the original rate reduction delay (300-500 μs). Our model captures the total congestion resolution latency as T_resolve = T_mark + T_response + T_recover, where for InfiniBand the retransmission term is zero (due to credit-based flow control) and for RoCE the retransmission term is approximately 200 μs per loss event. Over a 24-hour training run with 100 congestion events per hour, InfiniBand loses approximately 24 × 100 × 0.5 = 1,200 ms of training time to congestion resolution, while RoCE loses 24 × 100 × 0.7 = 1,680 ms—a 40% increase in congestion-induced training delay.