PFC Threshold & Config Generator
Precision calculator for L2 buffer requirements in lossless AI fabrics. Calculate XOFF/XON values and watchdog timers for Arista, Cisco, and NVIDIA-Mellanox switches.
Policy Parameters
Warning
Incorrect PFC configuration can cause **Head-of-Line Blocking** or **PFC Storms**. Always verify watch-dog timers (PFC-WD) are enabled.
! Arista EOS Configuration for RoCE v2
! Priority Flow Control (PFC) & Enhanced Transmission Selection (ETS)
!
interface Ethernet1-32
priority-flow-control mode on
priority-flow-control watch dog action errdisable
!
dcbx pfc 3
!
traffic-policy PFC-POLICY
class RDMA-CLASS
bandwidth percent 50
priority 3
class DEFAULT-CLASS
bandwidth percent 50
!
qos list pfc 3
Flow Control Logic
PFC enables per-priority pause frames. When the egress buffer of queue 3 reaches the XOFF threshold, a PAUSE frame is sent back to the upstream switch port.
Bandwidth Guarantee
ETS ensures that RDMA traffic (TC 3) receives at least 50% of the link capacity during congestion, preventing starvation from best-effort traffic.
1. The XOFF Threshold: Braking Distance Calculus
PFC is fundamentally about "Braking Distance." If a switch is full, and THEN sends a PAUSE frame, the bytes already in transit (the "In-Flight" data) will overflow the buffer.
XOFF Threshold Equation
For a 400Gbps port at 100 meters, the XOFF threshold must be roughly 25KB to accommodate the ~500ns in-flight persistence. Failure to account for cable length in hyperscale pods leads to intermittent RDMA Sequence Errors that are notoriously difficult to debug.
2. Congestion Spreading: The Viral PFC Paradox
The greatest risk of PFC is Head-of-Line Blocking (HoLB). If one GPU (Receiver A) is slow, the switch will PAUSE the sender. If that sender was also sending to a FAST receiver, that receiver is now starved.
Immediate Blocking
The slow port's queue fills, triggering a Layer 2 PAUSE signal back to the NIC. Intended local protection.
Horizontal Spread
The PAUSE signal propagates upstream to the Spine tier, eventually stopping unrelated training jobs on the other side of the datacenter.
3. The PFC Watchdog: Preventing Fabric Collapse
To prevent total collapse, modern switches implement a Watchdog. If a queue has been in a "PAUSE state" for more than 100ms, it is considered deadlocked.
Recovery Sequence Logic
Detection Criteria
ASIC registers the queue as a 'Victim' or 'Aggressor' based on PAUSE counter intervals.
The Hard Kill
The switch 'discards' all packets in the offending queue. While one flow fails, the thousands of others are released from the deadlock.
4. Industrial Design: Lossless Implementation Blueprint
Implementing PFC requires careful mapping of Class-of-Service (CoS) values to the "no-drop" hardware buffers.
CoS 3 Mapping
The industry standard for RDMA traffic. Dedicate a specific queue to ensure storage-class traffic never contends with best-effort web traffic.
DCQCN Strategy
Combining PFC (last resort) with ECN (proactive throttle) to ensure the network slows down before it has to stop. It is the golden standard for AI fabrics.
Buffer Headroom
Configuring separate 'Static' and 'Dynamic' buffer pools to prevent a single port from starving the entire packet buffer of the ASIC.
Frequently Asked Questions
Technical Standards & References
Related Engineering Resources
PFC Deadlock Physics: Headroom Calculation, Buffer Thresholds, and the 3-Phase Handshake
Priority Flow Control (PFC), defined in IEEE 802.1Qbb, operates as a per-priority pause mechanism at Layer 2. When a receiver's ingress buffer crosses a predefined XOFF threshold, it transmits a PFC pause frame to the sender, instructing it to cease transmission on that priority class for a duration specified in the pause quanta field (each quanta = 512 bit times, or ≈ 512 ns at 40 Gbps). The sender must stop transmitting within the "pause turnaround time," which includes the wire delay, the PFC frame serialization delay, and the link partner's processing delay. At 400 Gbps, a single bit time is 2.5 ps, and a maximum-size jumbo frame (9000 bytes) serializes in 180 ns. The link-level headroom required to absorb in-flight traffic during this turnaround is: H = T_turnaround × R_rate, where T_turnaround is the sum of media delay, PFC frame serialization time, and internal processing delay.
The XOFF and XON thresholds determine the efficiency of the PFC loop. XOFF is set sufficiently high to accommodate the headroom H plus a safety margin M, typically 1.5× H. XON is set below XOFF to create a hysteresis band that prevents oscillatory pause/unpause cycles (PFC toggling). For a 16 MB shared buffer port with eight priority classes, each priority receives a guaranteed 2 MB allocation. With H = 128 KB (a realistic value for a 400 Gbps link with 100m of fiber and a 50 μs cable delay), XOFF ≈ 192 KB, which is only 1.5% of the guaranteed buffer. However, when multiple priority classes experience congestion simultaneously, the dynamic buffer pool is shared across all congested priorities, and the headroom requirement scales linearly with the number of active PFC priorities. With all eight priorities active, the total headroom requirement is 8 × 128 KB = 1 MB, and the XOFF thresholds must be adjusted downward to prevent buffer exhaustion at the switch ASIC level.
The most dangerous operational failure mode is PFC deadlock, which occurs when two switches each wait for the other to drain a buffer, creating a cyclic buffer dependency (similar to a credit loop in PCIe). Deadlock manifests as a persistent state where both switches have issued PFC pause frames on the same priority in both directions, and neither can drain because neither is receiving traffic. The deadlock is resolved only by a link flap or a switch reboot unless the switch implements a deadlock detection timer that automatically drops paused frames after a configurable timeout (typically 500 ms to 2 seconds). Modern networks mitigate this using PFC watchdog timers that monitor the time elapsed since the last pause was received. If the pause duration exceeds a threshold, the watchdog preemptively drops packets to break the deadlock. Our config generator implements the RFC 8257 buffer headroom calculations and presents the XOFF/XON thresholds as tunable parameters, allowing operators to model the trade-off between buffer efficiency and deadlock probability before deployment.
PFC Watchdog Timers and Deadlock Detection Thresholds
PFC watchdog timers — implemented in most data center switch ASICs (Broadcom Tomahawk/Jericho, Mellanox Spectrum, Cisco Silicon One) as a safety mechanism against persistent PFC congestion — operate by monitoring the time elapsed since the last PFC pause frame was received on each port and priority class. When the watchdog timer exceeds a configurable threshold T_watchdog (typically 100-500 ms), the switch ASIC assumes that the remote link partner is in a PFC deadlock state and automatically drops all packets in the affected priority class's ingress buffer, clearing the buffer and allowing the PFC state machine to reset. The PFC deadlock detection threshold must be set high enough to avoid false positives during legitimate congestion bursts (where a busy storage target may pause the sender for 10-50 ms during a write burst) but low enough to detect genuine deadlocks before the application-layer TCP or RDMA timeout expires (typically 200 ms for the min RTO in TCP or 500 ms for the RC transport's responder timeout). A common production recommendation (per Cisco's MDS 9000 SAN-OS configuration guide and NVIDIA's InfiniBand/RoCE tuning guide) is to set T_watchdog = 200 ms for storage priorities (typically priority 3) and 100 ms for management priorities (typically priority 0). The PFC config generator's watchdog timer model computes T_watchdog_opt = min(T_application_timeout / 2, T_buffer_clear / 3), where T_buffer_clear is the time required to drain the priority's maximum buffer allocation at the slowest possible link speed (the port speed after a speed auto-negotiation fallback, not the max speed). For a priority with 4 MB of buffer allocation at a fallen-back link speed of 25 Gbps (one-third of the full 100 Gbps), the buffer clear time is T_buffer_clear = 4 × 10^6 × 8 / 25 × 10^9 = 1.28 ms — negligible compared to T_application_timeout / 2 = 100 ms, so T_watchdog_opt = 100 ms for storage priorities.
.The PFC quanta count — the duration encoded in the pause frame's time field, measured in units of 512 bit times — determines how long the sender stops transmitting after receiving the pause. The quanta value Q must be set such that Q × 512 / line_rate exceeds the maximum expected recovery time T_recovery, which is the time the receiver needs to drain its buffer below the XON threshold and prepare to accept new traffic. T_recovery depends on the egress link speed and the current buffer occupancy at the time the pause was issued. For a 100 Gbps link with 4 MB buffer occupancy at the XOFF trigger, the drain time at full egress speed is T_drain = 4 × 10^6 × 8 / 100 × 10^9 = 320 μs. The quanta should cover at least 2× T_drain to avoid repeated pause/unpause cycles: Q ≥ 2 × 320 μs / (512 / 100 × 10^9) = 2 × 320e-6 / 5.12e-9 = 125,000 quanta. The maximum quanta value in the IEEE 802.1Qbb standard is 65,535 (the 16-bit time field), so at 100 Gbps, the maximum pause duration is 65,535 × 512 / 100 × 10^9 = 335 μs — barely enough for a single drain cycle. At 400 Gbps, the maximum pause duration drops to 65,535 × 512 / 400 × 10^9 = 84 μs — far below the 320 μs drain time. This is the fundamental limitation of the PFC quanta field at high line rates: above 100 Gbps, the quanta field cannot encode a pause long enough to drain the buffer, and the receiver must send repeated pause frames (each after the previous pause expires) to keep the sender paused — increasing the probability of a pause frame loss and the consequent buffer overflow. The PFC config generator computes the required Q as a function of the line rate and the per-priority buffer allocation, and it flags configurations where Q exceeds the 65,535 maximum, recommending a larger per-priority buffer allocation (to reduce T_drain) or a higher link speed (to increase the quanta-per-microsecond ratio).
The interaction between PFC and the ETS (Enhanced Transmission Selection, IEEE 802.1Qaz) bandwidth allocation algorithm introduces a priority-based starvation risk that is distinct from the simpler pause-level deadlock. ETS allocates a minimum bandwidth percentage to each priority class: for example, priority 3 (storage) gets 50%, priority 4 (HPC) gets 30%, and priority 0 (management) gets 20%. When all three priorities are congested, ETS guarantees that each priority receives at least its allocated fraction of the egress link bandwidth. However, when PFC pauses priority 3 on the ingress port, the ETS algorithm recalculates the bandwidth distribution: the paused priority's allocated bandwidth (50%) is redistributed to the non-paused priorities proportionally (priority 4 gets 30% + 50% × 30/50 = 60%, priority 0 gets 20% + 50% × 20/50 = 40%). The non-paused priorities now have significantly more bandwidth, allowing them to fill the paused priority's buffer faster when the pause is released — because the receiver now has more free buffer space (the paused priority's buffer was drained during the pause), and the non-paused traffic can consume the freed buffer before the paused traffic can recover. This effect, called "PFC-induced buffer preemption," can cause the paused priority to experience repeated pause cycles (PFC toggling) even when its offered load is well below 50% of the total link capacity. The config generator's ETS interaction model simulates the three-priority ETS scheduler under PFC events and reports the "PFC toggle frequency" (cycles per second) for each priority. When the toggle frequency exceeds 100 Hz, the tool recommends reducing the ETS bandwidth allocation for the PFC-paused priority to prevent the preemption-induced toggling — a counterintuitive recommendation that improves throughput for the paused priority by reducing the tail-gating effect from other priorities.
"You are our partner in accuracy. If you spot a discrepancy in calculations, a technical typo, or have a field insight to share, don't hesitate to reach out. Your expertise helps us maintain the highest standards of reliability."
Contributors are acknowledged in our technical updates.