In a Nutshell

In the modern AI-scale data center, the decoupling of physical underlay and logical overlay via EVPN-VXLAN is mandatory for multi-tenant isolation and host mobility. However, this flexibility comes with a significant Byte Tax. Every frame originated by a host must be wrapped in a 50-byte matryoshka doll of headers, creating an MTU budget that, if miscalculated, will collapse throughput via CPU-driven fragmentation. This article deconstructs the Physics of Encapsulation, Route Type Logic, and the economics of Symmetric IRB in hyperscale fabrics.

BACK TO TOOLKIT

Fabric Efficiency & MTU Modeler

Analyze the mathematical overhead of VXLAN encapsulation. Simulate fragmentation risks and visualize the MTU ladder for your underlay/overlay.

Simulation Params

64MTU Standard (1500)Jumbo (9000)
15009216
Encapsulation Forensics

Packet Overhead Analysis

L2
IP
UDP
VX
L2
PAYLOAD (1400B)
Safe to Propagate
Efficiency: 0.00%
0B
Total Overhead
0B
Transit MTU
1500B
Remaining MTU
0.0%
Header Expansion
MTU vs. MSS Relationship

For TCP traffic traversing EVPN-VXLAN, the **MSS (Maximum Segment Size)** must be reduced to account for the encapsulation. If the absolute path MTU is 1500 bytes, the VXLAN overhead (typically 50 bytes) dictates a maximum IP payload of 1450 bytes. Subtracting the internal IPv4 and TCP headers (40 bytes), the ideal MSS should be set to **1410 bytes** to prevent performance-killing ICMP "Fragmentation Needed" events.

Jumbo Frame Necessity

In modern Leaf-Spine AI fabrics, implementing **Jumbo Frames (9000-9216 bytes)** on the underlay is mandatory. This provides sufficient "headroom" for nested encapsulation, multi-level VLAN tagging, and security headers while still allowing the standard 1500-byte client Ethernet frame to pass without fragmentation, significantly reducing CPU interrupts at the VTEP (Virtual Tunnel Endpoint).

Share Article

1. The Encapsulation Equation: The VXLAN Byte Tax

VXLAN (Virtual eXtensible Local Area Network) encapsulates Layer 2 frames into Layer 3 UDP packets. This allows Ethernet segments to span across a routed L3 underlay.

Packet Overhead Calculus

Louter=Linner+14⏟Eth+20⏟IP+8⏟UDP+8⏟VXLANL_{\text{outer}} = L_{\text{inner}} + \underbrace{14}_{\text{Eth}} + \underbrace{20}_{\text{IP}} + \underbrace{8}_{\text{UDP}} + \underbrace{8}_{\text{VXLAN}}
Ethernet (14B) | IP (20B) | UDP (8B) | VXLAN (8B)

The result is a 50-byte tax for IPv4 (74 for IPv6). If your underlay is restricted to a standard 1500-byte MTU, any 1500-byte guest frame will be fragmented into two packets, effectively doubling your packet-per-second (PPS) count and potentially crushing the destination CPU during reassembly.

2. IRB Architecture: Symmetric vs. Asymmetric

Integrated Routing and Bridging (IRB) defines how traffic moves between VNIs. Choosing the wrong model is the #1 cause of control-plane state bloat.

Symmetric IRB

Routing occurs at both source and destination VTEPs into a dedicated Transit VNI. High scalability—Leafs only need local VLAN configuration.

Asymmetric IRB

Ingress Leaf routes; egress Leaf only bridges. Requires every Leaf to carry state for EVERY VNI. Not recommended for fabrics larger than 10-15 nodes.

3. Route Type Forensics: The MP-BGP Core

EVPN differs from legacy VXLAN by using MP-BGP to advertise reachability. Understanding the five primary Route Types (RFC 7432) is critical for troubleshooting convergence.

Type-2: MAC/IP

The primary route for host reachability. Advertises both MAC and IP to enable ARP suppression at remote Leaf switches.

Scale∝Nhosts\text{Scale} \propto N_{\text{hosts}}
Type-1/4: ESI Logic

Ethernet Segment Identifiers enable multi-homing. Type-1 handles aliasing (ECMP), and Type-4 handles Designated Forwarder (DF) election.

ΔTfailover<50ms\Delta T_{\text{failover}} < 50\text{ms}

4. Industrial Blueprint: Zero-Fragmentation Fabrics

Building a hyperscale fabric requires rigid adherence to MTU and QoS standards. This is the Gold Standard for AI and Public Cloud infrastructure.

Universal 9216B MTU

Enabled across all physical Spine and Leaf interfaces. Eliminates the '50-byte trap' and allows for stacked NSH/Geneve headers.

Symmetric IRB Gateway

Uses Transit VNIs (L3VNI) for all inter-subnet traffic. Minimizes the required MAC-table size in hardware ASICs.

DSCP-to-Outer QoS

Copy internal RoCEv2 markings to the outer IP header. Ensures Spines respect lossless priority queues during congestion.

Frequently Asked Questions

Technical Standards & References

Mahalingam et al. (IETF)
RFC 7348: Virtual eXtensible Local Area Network (VXLAN)
VIEW OFFICIAL SOURCE
Sajassi et al. (IETF)
RFC 7432: BGP MPLS-Based Ethernet VPN (EVPN)
VIEW OFFICIAL SOURCE
Juniper Networks
EVPN-VXLAN Symmetric IRB Architecture Guide
VIEW OFFICIAL SOURCE
NVIDIA Networking
RoCEv2 over VXLAN: Performance and QoS Mapping
VIEW OFFICIAL SOURCE
Mathematical models derived from standard engineering protocols. Not for human safety critical systems without redundant validation.

Related Engineering Resources

Partner in Accuracy

"You are our partner in accuracy. If you spot a discrepancy in calculations, a technical typo, or have a field insight to share, don't hesitate to reach out. Your expertise helps us maintain the highest standards of reliability."

Contributors are acknowledged in our technical updates.

Share Article