The Gold Standard: Why InfiniBand XDR Still Rules the Training Floor

The fundamental architectural advantage of InfiniBand over Ethernet for AI fabrics is its **Credit-Based Flow Control (CBFC)**. Unlike Ethernet's lossless mode which relies on PFC (Priority Flow Control) pause frames, CBFC operates at the virtual lane level with per-destination credits that prevent buffer overflow without blocking unrelated traffic.

In CBFC, each receiver maintains a **Credit Counter** for every connected sender. When a sender transmits a packet, it decrements its credit balance for that destination. The receiver periodically returns credits via dedicated **Flow Control Packets** indicating how many bytes it has consumed from its receive buffer. If the sender runs out of credits, it must stop transmitting to that specific destination — but crucially, it can continue sending to other destinations whose credits are available. This eliminates the Head-of-Line blocking problem that plagues PFC-based RoCE networks.

The credit granularity depends on the InfiniBand generation. On XDR (800G) links, the credit unit is 64 bytes — the minimum VL (Virtual Lane) MTU. Each port maintains 16 independent credit pools, one per VL, allowing for 16 levels of traffic class isolation. Control traffic (like SMP subnet management packets) uses VL15, which has the highest priority and is never subject to credit starvation. This ensures that even under full data load, the fabric can always be reconfigured.

The deterministic nature of CBFC directly impacts training stability. In a 10,000-GPU cluster running All-Reduce, the slowest link determines the iteration time. RoCE v2 fabrics with PFC exhibit **credit stall propagation** where a single congested flow causes PFC pause frames to cascade across the entire fabric, freezing unrelated flows. InfiniBand's per-destination credit accounting prevents this cascade, maintaining predictable tail latency even at 95%+ link utilization. This determinism is why the largest AI clusters (Meta's 24,000-GPU RSC, Tesla's Dojo) exclusively use InfiniBand for their training fabric.

SHARP (Scalable Hierarchical Aggregation and Reduction Protocol) offloads the arithmetic reduction of All-Reduce and All-Gather from GPU Tensor Cores to the switch silicon itself. In a 32,768-GPU cluster, a standard ring All-Reduce requires each GPU to perform 32,767 data transfers (2N steps in a ring), consuming 100+ milliseconds of communication time per training step. SHARP reduces this to a logarithmic number of network traversals, with the switch hardware performing the addition or average operation as data passes through.

The latency scaling of SHARP is governed by the **Tree Depth** — the number of switch levels in the reduction tree. For a 2-tier Fat-Tree with leaf and spine switches, SHARP constructs a reduction tree with depth 2: leaf switches reduce data from their connected GPUs, then spine switches reduce the leaf outputs. The total reduction latency is 2 x (switch_latency + link_latency + reduction_compute_time). At XDR (800G) with 280 ns switch latency and 50 ns per meter of fiber (10-meter cables = 500 ns), each level adds 780 ns of latency. The total SHARP reduction for a 32,768-GPU cluster is 1.56 microseconds — compared to 100+ milliseconds for a ring All-Reduce of the same size.

The bandwidth scaling is equally favorable. In a ring All-Reduce, each GPU sends NCCL_BUFFSIZE amount of data per ring step. With SHARP, each GPU sends its data once to its leaf switch, paying the bandwidth cost exactly once regardless of the cluster size. The aggregate bandwidth requirement on the leaf-to-spine links is therefore proportional to the per-GPU data volume, not to the number of GPUs. For a 100 GB gradient tensor, the SHARP bandwidth on each leaf-to-spine link is 100 GB (one copy per leaf), while a ring All-Reduce would require 32,768 x 100 GB = 3.2 PB of aggregate bandwidth across all links. SHARP reduces the network bandwidth requirement by a factor of N (the number of GPUs), making it the only practical approach for training at the 100,000-GPU scale.

The key limitation of SHARP is the **Precision Constraint** — switch silicon uses fixed-point arithmetic for the reduction, and the accumulation of rounding errors limits the precision to FP16 or BF16 equivalents. FP32 gradients cannot be reduced in-network because the switch ALU lacks the dynamic range. For training loops that require FP32 gradient accumulation (common in mixed-precision training), SHARP reduces the FP16/FP16 partial gradients and the GPU performs the final FP32 accumulation as a post-processing step. The SHARP-v4 implementation in Quantum-3 supports FP8 in-network reduction, which will become the dominant mode for next-generation training where the master weights are stored in FP16 but the gradients are computed and reduced in FP8.