RDMA Throughput & Efficiency Predictor
A high-fidelity modeler for simulating RDMA goodput based on link attributes, delay, and protocol overhead.
Configuration
Accounting for one-way, fire-and-forget.
Round-trip time for completion.
RDMA Efficiency Breakdown
Wire efficiency and CPU savings for RDMA Write operations.
Packets Required
17
Goodput (MB/s)
48496.8
CPU Cycles Saved
3,342,336
"RDMA eliminates CPU from the data path. No memcpy, no context switches, no kernel crossings."
1. The Zero-Copy Mandate: CPU-Bypass Physics
To understand RDMA throughput, one must first understand the bottleneck it removes: the OS Kernel. In standard TCP, the probability of reaching 400Gbps on a single flow is near zero because the CPU must process every interrupt. RDMA moves the entire transport logic—segmentation, retransmission, and flow control—into the **NIC ASIC**.
Hardware-Offload Metrics
Because the CPU is bypassed, the throughput is limited only by the **HBM/DRAM** bandwidth of the server and the **PCIe Link** speed. On an NVIDIA H100 node, the RDMA NIC (ConnectX-7) can pull data from GPU memory at 400Gbps with zero impact on the training workload.
2. The BDP Constraint: Data in the Pipe
The Bandwidth-Delay Product (BDP) defines the amount of data required to fill the "pipe." If the sender stops before the first ACK/Credit arrives, the link goes idle.
BDP Equation
In RDMA, the "Window Size" is controlled by **Credits**. If the receiving NIC doesn't have at least 500KB of buffer space dedicated to this flow, the throughput will drop. In large-scale AI fabrics, managing these credits per-QP (Queue Pair) is the "Art of Fabric Optimization."
As distance increases (e.g., between data center zones), the RTT grows. For a 1km fiber link, the BDP for a 800Gbps fabric exceeds 2MB. Most standard NICs default to smaller credit pools, causing a major performance collapse over "long" distances unless tuned.
3. Credit-Based Logic: The Silence of the Wire
Unlike TCP, which uses a "dropping" signal to manage speed (Congestion Avoidance), InfiniBand and RoCE use a deterministic **Credit-Based** system.
Credit Advertisement
The receiver tells the sender: "I have space for 100 packets." The sender decrements this counter for every packet sent. When it hits zero, it stops—immediately. No dropped packets, no retransmissions.
Throughput Saw-Tooth
If credits are returned slowly (due to CPU stall at the receiver), the sender pulses. This creates a "Saw-Tooth" throughput pattern that is common in fragmented RDMA workloads.
4. Hardware Limits: The PCIe Gen5 Bottleneck
The network link is rarely the true bound. Modern 400G and 800G NICs are constrained by the server's internal architecture.
- PCIe Bandwidth Stealing
A Gen5 x16 slot provides 63GB/s (unidirectional). A 400Gbps NIC eats 50GB/s of that. If your system is also using PCIe for NVMe storage or other accelerators, the "contention" on the PCIe root complex will drop RDMA throughput by 10-15% as the bus negotiates priorities.
- Memory Latency (CAS)
RDMA Write is only as fast as the receiver's memory can sink it. If multiple flows hit the same memory bank simultaneously (Incast), the memory controller becomes a bottleneck, forcing the NIC to withhold credits and slowing the entire fabric.
5. Industrial Forensics: Auditing the Goodput
To measure actual goodput, you must look at the application-layer TFLOPS vs. the line-rate bits.
Frequently Asked Questions
Technical Standards & References
Related Engineering Resources
Credit-Based Flow Control Starvation
RDMA fabrics rely on credit-based flow control at the link layer. Each receiver grants a fixed number of credits to the sender, representing available buffer space. When credits are exhausted, the sender stalls — a condition that can cascade into fabric-wide starvation if not properly provisioned.
Credit Pool Sizing and Head-of-Line Blocking
Each RC (Reliable Connection) QP requires a minimum credit pool of credits to maintain full wire speed. For a 400 Gbps link with round-trip latency, credits. When multiple QPs share the same physical port, the total credit pool must be to avoid starvation.
VL Arbitration and Starvation Prevention
InfiniBand Virtual Lanes (VLs) provide independent credit pools that prevent starvation across traffic classes. Allocating a dedicated VL for All-Reduce traffic ensures that credit exhaustion on the storage VL does not stall collective operations. The weight-based arbitration between VLs () must be set to at least for latency-sensitive traffic. On RoCE fabrics, the equivalent is strict priority queues (802.1p) with PFC enabled only on the highest priority class to avoid the credit starvation cascade.
PCIe Gen5/Gen6 Link Utilization and TLP Header Overhead in RDMA Transfers
The PCIe link between the GPU and the RDMA NIC is the first serialization bottleneck in any RDMA transfer. Even if the network link runs at 400 Gbps (50 GB/s), the PCIe Gen5 x16 link provides 64 GB/s of raw bandwidth—only 28% headroom. However, the PCIe transaction layer adds significant overhead that reduces effective throughput. Each Memory Read (MRd) or Memory Write (MWr) transaction layer packet (TLP) must carry a 12-byte or 16-byte header that includes the transaction descriptor (address, length, attributes), a 4-byte digest (ECRC) when enabled, and potentially the 4-byte prefix for TLP Processing Hints (TPH). For a 64-byte cacheline-aligned RDMA write, the TLP overhead is: 16 bytes (header) + 64 bytes (data) + 4 bytes (ECRC optional) = 84 bytes total, of which 64/84 = 76.2% is payload. For a 256-byte write, the efficiency improves to 256/(256+16+4) = 92.8%. The NVLink-C2C interconnect used in Grace Hopper superchips further compounds this: NVLink-C2C encapsulates PCIe TLPs within a proprietary link-layer frame that adds 6 bytes of link-layer control per 256-byte flit, reducing the effective PCIe throughput to 88% of the raw Gen5 x16 rate, or 56.3 GB/s effective—only 12.5% headroom above the 50 GB/s 400 Gbps network rate.
The max payload size (MPS) negotiation during PCIe link training determines the maximum data payload per TLP. Most RDMA NICs negotiate MPS = 256 bytes (the Gen5 default), but some NICs support MPS = 512 or 1024 bytes on Gen5. Increasing MPS from 256 to 512 bytes reduces the header overhead fraction from 20/276 = 7.25% to 20/532 = 3.76%, improving effective throughput by 3.5%. However, larger MPS increases the blocking latency for higher-priority traffic on the same PCIe root port. A 512-byte TLP occupies the PCIe Gen5 x16 lane for 512 × 8 / (32 × 25 × 10⁹) = 5.12 ns, blocking any interrupt-driven completions (CQEs) from the NIC that must traverse the same root port. For RDMA operations where completion notification latency directly affects the next work request posting, this blocking adds approximately 1-3 PCIe clock cycles (40-120 ps) per TLP, which is negligible at the microsecond scale of RDMA latency. Our throughput model includes a PCIe TLP efficiency factor η_pcie = MPS / (MPS + H_TLP) where H_TLP = 20 bytes (header + optional ECRC), allowing the engineer to determine the optimal MPS for their specific NIC and GPU combination.
The PCIe read completion boundary (RCB) constraint affects RDMA Read operations that must converge with the network read response. When the NIC issues a PCIe Memory Read (MRd) to read GPU memory for an RDMA Send, the GPU memory controller returns the data in completions (Cpl) that must align to the GPU's cacheline boundary (128 bytes for H100, 64 bytes for A100). If the NIC's MRd request address is not cacheline-aligned, the GPU controller must split the completion into two separate Cpl TLPs, doubling the completion overhead. RDMA NICs that support PCIe's Read Completion Boundary (RCB) parameter (set to 64 or 128 bytes) can issue aligned requests, avoiding the split. Our model verifies that the NIC and GPU RCB settings match and adds a 1.5× overhead multiplier when they are misaligned. At 400 Gbps with 4 KB RDMA messages, an RCB mismatch reduces goodput by 3-5% due to the increased TLP completion count on the PCIe bus.
The PCIe Gen6 (PAM-4) transition, expected in 2026-2027 for 800 Gbps NICs, doubles the per-lane data rate from 32 GT/s (Gen5) to 64 GT/s using PAM-4 signaling. However, PAM-4 reduces the voltage margin by 3× compared to NRZ (Gen5), requiring the NIC to implement a more complex decision-feedback equalizer (DFE) in the PCIe PHY. The DFE adds 2-3 ns of link wake-up latency for power-state transitions (L1 substate) that can impact the NIC's ability to react to bursty RDMA traffic patterns. Our model simulates the PCIe Gen6 link efficiency as η_gen6 = 1 - (PAM4_overhead / (MPS + H_TLP)), where PAM4_overhead accounts for the additional Reed-Solomon FEC (RS-FEC) that PCIe Gen6 adds at the physical layer. RS-FEC in PCIe Gen6 uses RS(544,514) encoding that adds 30 symbols of overhead per 544-symbol block (5.5% overhead), reducing the effective data rate from 64 GB/s to 60.4 GB/s for a x16 link—still 20% headroom above the 50 GB/s required for 400 Gbps, but only 7% headroom above the 100 GB/s required for 800 Gbps. Our model alerts the operator when the PCIe link efficiency drops below 1.15× the network line rate, indicating a mismatch between the PCIe generation and the NIC speed that will bottleneck RDMA throughput.
"You are our partner in accuracy. If you spot a discrepancy in calculations, a technical typo, or have a field insight to share, don't hesitate to reach out. Your expertise helps us maintain the highest standards of reliability."
Contributors are acknowledged in our technical updates.