Managing Lossless Ethernet

Priority Flow Control (PFC) deadlocks are the single most destructive failure mode in RoCE-based AI fabrics. A PFC deadlock occurs when two switches are each waiting for the other to drain their receive buffers before resuming transmission, creating a circular dependency that freezes all traffic on the affected priorities. Unlike TCP's timeout-triggered retransmission, PFC has no built-in deadlock recovery mechanism — once paused, a port will remain paused until it receives an XON frame, which never arrives if both sides are waiting.

The root cause of PFC deadlocks is **Credit Starvation** in multi-hop topologies. Consider three switches A, B, C in a spine-leaf-leaf topology. Switch A (leaf, transmitting to leaf C via spine B) sends 9000-byte jumbo frames to B. Switch B's egress port to C becomes congested and B sends a PFC XOFF to A for priority 3. A stops transmitting priority 3 traffic to B — but A's egress buffer to B is now filled with priority 3 packets that cannot be drained because B's pause has frozen A's transmission. If A was also receiving priority 3 traffic from C (via B) at the same time, C's egress buffer to B fills with packets destined for A, and B sends XOFF to C. Now A is waiting on B, B is waiting on C, and C is waiting on A — a classic circular wait that no switch can resolve independently.

Modern data center switches implement **PFC Deadlock Detection (PDD)** using a watchdog timer per priority per port. If a port remains in the paused state (XOFF received, no XON received) for longer than the **PFC Deadlock Detection Timer** (default 250 ms on Spectrum-4), the switch assumes a deadlock and enters **Deadlock Recovery Mode**. The recovery mode forcibly flushes the paused port's receive buffer — dropping all packets in the buffer regardless of their priority or destination. After flushing, the switch sends an XON frame to the neighbor, breaking the circular dependency. The dropped packets are recovered by the RDMA transport layer (Go-Back-N retransmission), which is slower than PFC drop-free operation but far better than an infinite freeze.

The PDD timer value is a critical tuning parameter. If set too short (below 50 ms), normal congestion events during All-Reduce — where a port may legitimately remain paused for 20-30 ms — trigger false deadlock detection, causing unnecessary packet drops that degrade throughput by 5-10%. If set too long (above 1 second), a genuine deadlock freezes the fabric for a full second, during which 50,000+ training iterations are stalled. The recommended value for AI training fabrics is 500 ms — long enough to distinguish between normal congestion and deadlock, short enough to limit training disruption to under 500 milliseconds. Coupled with randomized CNP jitter (which prevents the synchronous rate reductions that lead to deadlocks), the PDD mechanism reduces deadlock incidence to fewer than one event per 10,000 hours of cluster operation in well-tuned 800G fabrics.

PFC only delivers zero-loss operation if the receiving switch has enough buffer to absorb every frame that is in flight when the PAUSE frame is sent. The governing equation is B_req = 2 x R x (T_prop + T_pause_process) + MTU_max, where R is the link rate, T_prop is the one-way propagation delay, T_pause_process is the time for the peer to decode and act on the PAUSE, and MTU_max covers the frame that was already mid-transmission. For a 500-meter fiber at 5 ns per meter, T_prop is 2.5 microseconds each way; over a 400 Gbps link, one direction of propagation alone represents 400 Gbps x 2.5 us = 1,000,000 bits, or 125 kilobytes. Double that for the request-response nature of the handshake, add PAUSE processing latency and a full jumbo frame, and a conservative XOFF threshold on a 400G RoCE port lands between 250 and 300 kilobytes per priority per port — an order of magnitude above the 3,000-byte minimum the IEEE 802.1Qbb standard mandates.

The distance term is the one operators forget when they copy a reference configuration. An intra-rack 3-meter copper run contributes essentially nothing (T_prop = 15 ns, or 6,000 bits at 400G), but a 1,500-meter fiber path across a building contributes 7.5 microseconds each way — 750 kilobytes of in-flight data per direction at 400G. The same threshold that is generous on a short spine-leaf link becomes dangerously shallow on a long-distance inter-rack path, and the fabric silently begins dropping frames exactly when a tail-heavy training burst arrives. This is one reason the jumbo frame decision and the PFC threshold decision must be made together: raising MTU from 1,500 to 9,000 bytes raises the in-flight requirement by exactly the frame-size delta, and switches with fixed per-port buffer pools trade off headroom against frame granularity.

Threshold placement is then a question of how much headroom you convert into pause-absorption time. With a 300 KB per-priority pool and XOFF placed 90 KB below capacity, that headroom absorbs 90,000 x 8 / 400 Gbps = 1.8 microseconds of sustained line-rate burst before egress must be paused — matching the round-trip PAUSE reaction on a medium-distance link. Because absorption time scales as headroom divided by link rate, an 800G port needs twice the headroom for the same reaction window, which is why 800G fabrics ship with substantially deeper per-port buffer pools than their 400G predecessors. The absorption window must always exceed the longest configured link's round-trip reaction time, or the PAUSE arrives too late and the drop happens anyway.

Finally, remember that PFC is negotiated per traffic class, so each class needs its own sized pool. A fabric that runs RoCE (priority 3), storage (priority 4), and management (priority 0) with different MTUs must size three independent XOFF/XON pairs — and the pools compete for the same physical SRAM. Every kilobyte dedicated to a management class is a kilobyte unavailable to absorb an RDMA burst, which is why the guidance is to enable PFC only on the classes that carry lossless traffic and let everything else run best-effort. The interaction between these per-class budgets and the RoCE transport they protect is exactly what the queueing math has to optimize before the fabric goes live.

PFC and ETS settings do not propagate by magic — they are negotiated across the link by Data Center Bridging eXchange (DCBX), defined in IEEE 802.1Qaz and carried inside LLDP. Every link exchanges three relevant TLVs: the ETS Configuration TLV (the bandwidth percentages per priority), the ETS Recommendation TLV (suggested scheduling hints for devices that cannot honor explicit percentages), and the PFC TLV (which priorities are lossless). The willingness model is asymmetric: a port declares itself willing to accept the neighbor's configuration, or sends its own and expects the neighbor to comply. Getting this wrong in either direction — both ends willing, or neither willing — produces silently divergent priority-to-queue mappings and is the single most common cause of "lossless on paper, dropping in practice" RoCE fabrics.

ETS itself is implemented with a weighted scheduling algorithm, typically Deficit Weighted Round Robin (DWRR) with a per-cycle credit budget. On a 400G link with a 1-microsecond scheduling cycle, the scheduler issues credits equal to the link rate times the cycle time (400,000 bits, or 50 kilobytes) distributed across the bandwidth groups in proportion to their weights. A class weighted at 80% receives 320,000 bits of credit per cycle; because packet sizes rarely divide the credit evenly, the deficit counter carries the remainder into the next round. The consequence is that the 80/20 split is honored over hundreds of microseconds, not in any single cycle — which matters when a burst of 9,000-byte frames arrives: a priority weighted at 20% cannot transmit a frame larger than its accumulated credit without borrowing, and the borrow is repaid from its future allocation.

Multi-vendor fabrics add the final layer of risk because defaults diverge. NVIDIA Spectrum, Cisco Nexus, and Arista deliver different default priority-to-queue maps, different PFC watchdog behaviors, and different interpretations of the willing bit. The safe pattern is to make one device the configuration source (willing=false, sending authoritative TLVs) and every other device willing to learn — usually the ToR acting as the leaf is the source and the host NICs are the learners. Operators also need to verify the trust boundary: DCBX normally arrives on dot1p priority tags, so hosts that strip 802.1Q tags (or NICs that trust DSCP instead) silently bypass the negotiated profile. Auditing a fresh fabric means checking that every port shows the same ETS profile, the same PFC-enabled priority set, and the same watchdog settings — a task you can quantify end-to-end with the RDMA throughput predictor before committing a workload.

The interaction between DCBX and the underlying lossless fabric also surfaces at reconfiguration time: every DCBX change triggers a momentary pause on the affected priorities while the new mapping is applied, so rotating bandwidth profiles during a live training run introduces a controlled 10-50 millisecond blip per link. Batch these changes into maintenance windows, apply them from a single source of truth, and validate with pause-count telemetry on both ends of every link — sustained asymmetry in pause counters between peers is almost always a DCBX mismatch rather than a traffic problem. Where DPUs are present, the same profile is typically applied in the DPU offload layer, which means the negotiation now spans three trust domains instead of two and deserves an explicit conformance test.