RDMA atomic operations—Compare-And-Swap (CAS) and Fetch-And-Add (FAA)—are critical for implementing distributed locks, atomic counters, and collective synchronization primitives in GPU clusters. Unlike regular RDMA Reads and Writes, atomic operations require the target NIC to execute a read-modify-write operation on its local memory, then return the original value to the requester. This round-trip operation adds two full network traversals (request + response) plus the NIC's local memory access latency. On a ConnectX-7 400 Gbps InfiniBand NIC, the atomic operation latency is composed of: PCIe round-trip to the NIC's on-chip SRAM (approximately 0.5-1 μs), the NIC's atomic execution engine processing time (0.2-0.5 μs for 64-bit CAS), and two network traversals at 100 μs RTT (50 μs each direction), for a total of approximately 101-102 μs. This is 50× slower than a local atomic operation (approximately 2 μs for x86 LOCK CMPXCHG on 64-byte cacheline). The throughput of RDMA atomics is limited by the NIC's atomic operation pipeline depth—ConnectX-7 supports up to 16 concurrent atomic operations per QP, yielding a maximum throughput of 16 / 102 μs = 157,000 operations/second per QP. With 128 QPs (8 NICs × 16 QPs per NIC), the aggregate throughput is 128 × 157,000 = 20 million atomic operations/second—sufficient for fine-grained distributed locking in 1,000-GPU training clusters where each GPU performs an atomic synchronization once per training step at 1,000 steps/second.

The cacheline alignment constraint for RDMA atomics is more restrictive than for regular RDMA operations. The atomic operation target address must be aligned to the atomic operand size (8 bytes for 64-bit CAS/FAA on ConnectX-7), and the target memory must reside within a single 64-byte cacheline. If the target address crosses a cacheline boundary, the NIC must perform two separate atomic operations, doubling the latency and halving the throughput. The NIC's atomic engine does not support cross-cacheline atomics natively; instead, the NIC raises a protection error (IBV_WC_REM_OP_ERR) and the software must implement a multi-step fallback using RDMA Read-Modify-Write sequences. Our overhead model checks the alignment of the target address against the NIC's atomic alignment requirement and raises a warning when the target spans multiple cachelines. The performance impact of misalignment is modeled as: T_misaligned = T_atomic + N_cachelines × (T_RDMA_Read + T_RDMA_Write + T_CPU_atomic), where N_cachelines = 2 for a cross-cacheline atomic and T_RDMA_Read = 100 μs (including PCIe RTT), T_RDMA_Write = 50 μs (one-way), and T_CPU_atomic = 2 μs. This gives T_misaligned = 102 + 2 × (100 + 50 + 2) = 406 μs—a 4× increase over the aligned atomic operation, which can significantly impact distributed lock performance in large-scale training.

The memory ordering consistency model of RDMA atomics differs from the CPU's memory model, creating subtle correctness issues for distributed synchronization algorithms. RDMA atomics provide per-QP ordering: all atomic operations within the same QP complete in the order they were posted, and all RDMA Writes to the same target memory region that are posted before an atomic are guaranteed to complete before the atomic executes. However, there is no ordering guarantee across different QPs or between RDMA Read completions and atomic operations. This means that a distributed locking algorithm using CAS to acquire a lock on Node B must ensure that the lock variable is written to Node B's memory via an RDMA Write before the CAS operation is posted to the same QP—otherwise, the CAS may observe stale data. The standard implementation posts the lock request (RDMA Write to the lock variable) followed by the CAS (to atomically acquire the lock) in the same QP's work queue, relying on the per-QP ordering guarantee. Our overhead model verifies that the atomic-to-write ordering constraint is satisfied by checking the QP assignment: if the CAS and the preceding RDMA Write use different QPs, the model flags a potential ordering violation that could cause a distributed lock failure (two nodes simultaneously believing they hold the lock).

The RoCE atomic support status is a commonly overlooked constraint in RoCE-based GPU clusters. RoCE v2 (RFC 5042) defines atomic operations as optional, and many commodity RoCE NICs (e.g., Broadcom NetXtreme-E series, Intel E810) do not implement hardware atomic support, returning IBV_WC_NOT_SUPPORTED for any atomic work request. In such cases, the application must fall back to a software-emulated atomic using RDMA Read-Modify-Write: the client reads the target 64-byte cacheline via RDMA Read, performs the CAS or FAA in software on its local CPU, and writes the result back via RDMA Write with a conditional guard (a separate RDMA CAS on a guard variable). The software-emulated atomic latency is T_sw_atomic = T_RDMA_Read + T_CPU_atomic + T_RDMA_Write + T_RDMA_CAS_guard = 100 + 2 + 50 + 102 = 254 μs—250% overhead compared to the hardware atomic (102 μs). Our overhead model probes the NIC's atomic capability during initialization and selects the optimal atomic path (hardware if available, software-emulated if not) while reporting the throughput impact to the user. For clusters that require high-performance distributed locking (e.g., NCCL's topology discovery and collective scheduling), the model recommends InfiniBand NICs (which universally support hardware atomics) over RoCE NICs (where support is vendor-dependent).