What is ECN (Explicit Congestion Notification)?

ECN is a mechanism that allows switches to signal congestion to end-hosts by marking bits in the IP header, rather than dropping packets. This allows the sender to reduce its transmission rate before a buffer overflow occurs, maintaining 'lossless' performance.

What are Kmin and Kmax in ECN?

Kmin is the lower buffer threshold; below this level, no packets are marked. Kmax is the upper threshold; above this level, every packet is marked. Between Kmin and Kmax, packets are marked with a linear or exponential probability controlled by Pmax.

Why is ECN critical for RoCE v2?

RoCE v2 relies on Priority Flow Control (PFC) to prevent loss. However, PFC is a 'blunt' tool that stops all traffic on a priority level. ECN provides a 'surgical' response, slowing down only the specific flows causing congestion, thus preventing PFC-induced deadlocks and Head-of-Line blocking.

DCQCN (Data Center Quantized Congestion Control) is the industry-standard algorithm for RoCE v2. it uses ECN markings as a feedback signal to adjust the hardware rate limiters (RL) on the NIC, ensuring the fabric stays in a high-throughput, low-latency 'sweet spot'.

How do I calculate the ideal Kmin?

Kmin should be set high enough to allow for small bursts of traffic (to avoid unnecessary rate limiting) but low enough to trigger before the PFC 'XOFF' threshold is reached. A common rule of thumb is to set Kmin to 20-30% of the target queue size.

BACK TO TOOLKIT

ECN & DCQCN Threshold Modeler

A precision simulator for congestion control in lossless fabrics. Optimize your Kmin, Kmax, and Pmax parameters to maximize AI training goodput.

Fabric Topology

Link Speed400 Gbps

Fabric RTT (µs)10 µs

Ingress Buffer (MiB)32 MiB

Congestion Proto

K_min (Start Marking)

150KB

Threshold where CE bits start being marked in packet headers.

K_max (Full Marking)

450KB

Threshold where 100% of packets are marked for proactive throttling.

DCQCN Probability Curve

Calculated marking probability for a 400G line-rate congestion event.

0% PROB

100% PROB (DCQCN)

K_MIN

K_MAX

BDP (Giant)

488.3 KB

Buffer Usage

1.4%

XOFF Safety

Headroom: 32318 KB

"Optimal ECN ensures throughput remains at 99.9% while preventing PFC pauses from ever firing."

1. The Marking Probability: Modeling Buffer Occupancy

ECN does not work like a toggle; it is a probabilistic engine. The goal is to provide a "Soft Brake" that scales in intensity as the queue grows.

Linear Marking Equation

P(mark) = \begin{cases} 0 & \text{if } q < K_{min} \\ \frac{q - K_{min}}{K_{max} - K_{min}} \cdot P_{max} & \text{if } K_{min} \leq q \leq K_{max} \\ P_{max} & \text{if } q > K_{max} \end{cases}

q: Current Queue Depth | Pmax: Max Probability (20-100%)

By setting $K_{min}$ too low, we trigger rate limiting prematurely, hurting throughput. By setting it too high, we risk hitting the switch ASIC's physical buffer limit, triggering **PFC PAUSE** which kills performance for all flows.

2. DCQCN: The RDMA Feedback Loop

Standard ECN was designed for TCP. **DCQCN** is a specialized version for hardware-accelerated RDMA (RoCE v2).

The Feedback (CNP)

When the receiver receives a packet with the ECN bit set (CE=11), it calculates a 'Congestion Notification Packet' (CNP) and sends it back to the sender. This 64-byte frame is the "Brake Pedal" signal.

The Rate Limiter (RL)

The sender's NIC receives the CNP and instantly clamps the hardware rate limiter for that specific flow. It then slowly 'probes' for higher bandwidth (Additive Increase) until it sees another CNP.

3. The PFC Collision Course: Kmax vs. XOFF

The ultimate goal of ECN tuning in an AI fabric is to ensure that ECN rate limiting happens **before** PFC flow control is triggered.

Threshold Strategy

1. **Kmax < PFC XOFF**: If Kmax is reached, the router is marking 100% of packets, signaling a 'Hard Brake' to the senders.
2. **PFC XOFF**: If the buffer continues to grow, PFC sends a PAUSE frame. This is a cluster-wide 'Emergency Stop' that causes Jitter.
3. **Best Practice**: Set Kmax significantly lower than the PFC headroom to allow the DCQCN algorithm time to stabilize the flows before the fabric locks up.

4. Industrial Tuning: The Optimization Matrix

Tuning ECN is highly dependent on your link speed and target "Braking Distance."

Frequently Asked Questions

Technical Standards & References

Zhu, Y. et al. (Microsoft Research)

Congestion Control for Large-Scale RDMA Deployments (DCQCN)

VIEW OFFICIAL SOURCE

Ramakrishnan, K. and Floyd, S.

RFC 3168: The Addition of Explicit Congestion Notification (ECN) to IP

VIEW OFFICIAL SOURCE

NVIDIA Networking

NVIDIA RoCE v2 Congestion Management Guide

VIEW OFFICIAL SOURCE

Arista Networks

Arista: Configuring DCQCN on 7060X/7260X series

VIEW OFFICIAL SOURCE

Mathematical models derived from standard engineering protocols. Not for human safety critical systems without redundant validation.

Related Engineering Resources

Interactive Tool

PFC Config Generator

Coordinate ECN with flow control thresholds.

Interactive Tool

RoCE Overhead Calculator

Map congestion signals to RDMA framing.

Interactive Tool

Collective Ops Modeler

Model the impact of ECN on sync times.

Partner in Accuracy

"You are our partner in accuracy. If you spot a discrepancy in calculations, a technical typo, or have a field insight to share, don't hesitate to reach out. Your expertise helps us maintain the highest standards of reliability."

Contributors are acknowledged in our technical updates.

ECN
Thresholds.

In a Nutshell

ECN & DCQCN Threshold Modeler

Fabric Topology

DCQCN Probability Curve

1. The Marking Probability: Modeling Buffer Occupancy

Linear Marking Equation

2. DCQCN: The RDMA Feedback Loop

The Feedback (CNP)

The Rate Limiter (RL)

3. The PFC Collision Course: Kmax vs. XOFF

Threshold Strategy

4. Industrial Tuning: The Optimization Matrix

Frequently Asked Questions

Technical Standards & References

Related Engineering Resources

PFC Config Generator

RoCE Overhead Calculator

Collective Ops Modeler