MTU 9000: Why Jumbo Frames are Mandatory
The 1500B Bottleneck
The standard Ethernet Maximum Transmission Unit (MTU) of **1500 bytes** was established decades ago for 10Mbps networks. At 400Gbps, transmitting a 1GB file in 1500B chunks requires more than **700,000 packets**. Each packet generates an interrupt, a CPU context switch, and a header lookup.
Interrupt Coalescence
Fewer packets mean fewer hardware interrupts handled by the CPU. This frees up cycles for the actual AI training workload or NCCL collective operations.
Header Efficiency
Header overhead (L2+L3+L4) is constant per packet. A larger MTU increases the payload-to-header ratio, significantly improving goodput in RDMA-intensive fabrics.
Comparison: 1500B vs. 10240B MTU
| Metric | 1500B MTU | 9000B (Jumbo) |
|---|---|---|
| Packets per 1GB Data | ~715,000 | ~115,000 |
| CPU Context Switches | High (Bottleneck) | Minimal |
| Throughput Stability | Erratic | Sustained Peak |
| Switch Buffer Usage | Optimized for small packets | Requires careful tuning |
Performance Trade-offs
While Jumbo Frames are a net gain for bulk data transfer (AI training), they introduce "Head of Line Blocking" for smaller status packets. In high-performance AI fabrics, this is mitigated through **ETS (Enhanced Transmission Selection)**, which prioritizes small control packets over large data frames.