Ultra
Ethernet.

Traditional InfiniBand and RoCE v2 use a retransmission model where a single lost packet forces the sender to retransmit all subsequent packets (Go-Back-N). This is catastrophic at 800G line rates, where a single packet loss can stall the pipeline for microseconds, losing gigabytes of throughput. The Ultra Ethernet Consortium (UEC) has designed a fundamentally different reliability model.

UEC introduces **Selective Repeat with Out-of-Order Delivery**. Each packet carries a unique **Packet Sequence Number (PSN)** in the UEC transport header. The receiver maintains a **Bitmask Accumulator** of received PSNs. When a gap is detected (expected PSN 100, received PSN 102), the receiver immediately sends a **Selective NACK (SNACK)** containing the missing PSNs. Crucially, the receiver does not stall its delivery pipeline — packets 102, 103, and 104 are forwarded to the application while waiting for the retransmission of packet 101.

The sender handles the SNACK by consulting its **Transmission History Buffer**, a circular ring buffer in on-chip SRAM that stores the last 4,096 transmitted packets. It locates packet 101 and retransmits it at a higher priority, bypassing the normal Traffic Shaper queue. This **Priority Retransmission** path ensures the missing packet arrives within a single round-trip time, rather than waiting behind new data. The UEC specification mandates a maximum retransmission latency of 2.5 microseconds for an 800G link.

Beyond packet-level retransmission, UEC introduces **Path-Layer FEC (PL-FEC)**. Unlike Ethernet's RS-FEC which operates on physical codewords, PL-FEC operates on UEC packets themselves. The sender groups every 64 packets into a **FEC Block** and appends 8 parity packets computed using a Cauchy Reed-Solomon matrix. The receiver can recover any 8 missing packets within a block without requesting retransmission. This eliminates the round-trip latency of SNACK for up to 12.5% packet loss, which is critical for AI training clusters where congestion-induced micro-bursts are the norm.

Packet Spraying is the UEC feature that most directly impacts AI training throughput. Unlike ECMP's flow-level hashing, UEC's per-packet spraying distributes individual packets of a single flow across all available equal-cost paths in the fabric. In a spine-leaf topology with 8 spine switches, a flow of 64 packets is distributed as 8 packets per spine, regardless of the hash of the 5-tuple. This eliminates the "elephant flow collision" problem where two large All-Reduce flows hash to the same spine and share 400 Gbps instead of each getting 800 Gbps. The spraying is performed by the NIC's **Path Selection Engine (PSE)** , which maintains a real-time **Congestion Vector** of per-port utilization across all 8 paths.

The PSE uses a **Weighted Random Spray** algorithm. Each spine port is assigned a weight inversely proportional to its current queue depth (as reported by the switch's INT telemetry). A port with queue depth 10 KB gets weight 0.8, while a port with queue depth 100 KB gets weight 0.08 — the congested port is 10x less likely to be selected for the next packet. The weights are normalized and the packet is assigned to a port via a weighted random draw. This probabilistic approach avoids the synchronization problem of deterministic round-robin, where flows from different senders can collide at the same spine if they happen to be in lockstep.

The fundamental challenge of packet spraying is **Reordering at the Receiver**. Because packets take different paths with different latencies, they arrive at the destination out of order. The receiver's **Reassembly Buffer** must reorder packets into the correct sequence before delivering data to the RDMA consumer. The reassembly buffer size is the product of the maximum path latency differential and the line rate. In a well-balanced fabric, the maximum latency differential between the shortest and longest path is 150 nanoseconds. At 800 Gbps, this corresponds to 150 ns x 100 GB/s = 15 KB of buffered data per flow. For 1,024 concurrent flows, the total reassembly buffer requirement is 15 KB x 1,024 = 15 MB of on-chip SRAM per NIC port.

The UEC specification mandates a **1 MB reassembly buffer per NIC** as a minimum, which supports 68 concurrent flows with full worst-case reordering. For AI training workloads where the NCCL library opens 64 concurrent QPs (one per peer GPU in a 64-GPU job), the 1 MB buffer provides adequate coverage for the first 64 flows but leaves no headroom for management traffic. Higher-end UEC NICs implement 4 MB of reassembly buffering, supporting 273 concurrent flows and providing 400% headroom for flow bursts during the All-Reduce startup phase where all 64 QPs are simultaneously active. The buffer is implemented as a 32-way set-associative cache indexed by the flow's UET session ID, with a least-recently-used (LRU) eviction policy for flows that exceed the buffer capacity.