1. The Death of North-South: The East-West Surge
In the legacy data center, traffic was **North-South**. A user (North) requested a file from a server (South). Today, 80%+ of traffic is **East-West**. A single user request hits a Web server, which then talks to 50 Microservices, 10 Databases, and 2 caches (East-West) before replying.
The Clos Fabric Axiom
Traditional 3-tier models (Core-Agg-Access) break under East-West traffic because packets must 'hairpin' up to the Core and back down, creating massive latency. The Clos (Spine-Leaf) model solves this by making every server exactly the same distance from every other server.
Non-Blocking ECMP
Traffic is hashed across all spines. If you have 4 spines, you have 4 parallel highways. If you add a 5th spine, the capacity of the entire data center increases linearly.
Deterministic Latency
Whether the server is in the next rack or on the other side of the building, every packet traverses exactly 3 hops (Leaf → Spine → Leaf).
2. VXLAN & EVPN: The Binary Overlay
In a cloud, a Virtual Machine (VM) must be able to move racks without losing its IP. We use **VXLAN** (L2-over-L3 encapsulation) to 'stretch' the network, and **BGP-EVPN** to act as the brain that tracks where every MAC address lives.
VXLAN VTEP Forensics
BGP-EVPN Route Type-2 Forensics
Instead of broadcasting ARP requests, the network uses BGP to say: 'MAC AA:BB is at VTEP 10.1.1.5'. This is **Control Plane Learning**, and it's what allows clouds to scale to millions of endpoints without broadcast storms.
3. RoCE v2 & RDMA: Bypassing the CPU
Standard TCP/IP is too slow for AI/ML training. The latency of the OS kernel and CPU handling packets is the bottleneck. **RDMA** (Remote Direct Memory Access) allows a GPU to read data from a remote storage node directly, with **Zero-Copy** overhead.
The RoCE v2 Stack
RoCE v2 encapsulates RDMA frames into UDP/IP, allowing it to run over standard high-performance Ethernet fabrics. However, it requires a Lossless Network using Priority Flow Control (PFC) to ensure packets are never dropped, as RDMA has no built-in retransmission like TCP.
Performance Forensics:
Under load, a standard 100G TCP link might see 25% CPU overhead and 50microsecond latency. RoCE v2 reduces CPU overhead to <1% and latency to <1microsecond. This is the difference between an AI model training in 3 weeks vs. 3 days.
4. PUE & Thermal Forensics: The Physics of Cooling
A data center isn't just a network; it's a massive thermodynamic challenge. We measure efficiency using **PUE (Power Usage Effectiveness)**.
The Efficiency Math
A PUE of 2.0 means for every watt used by a server, a watt is wasted on cooling. Hyperscalers (Google/Meta) achieve PUEs as low as 1.07 using evaporative cooling and advanced Hot-Aisle Containment, where the exhausted hot air is physically sealed away from the cold air intakes.
5. NVMe-over-Fabrics: The End of the SCSI Bottleneck
For decades, network storage relied on the SCSI protocol (iSCSI/Fiber Channel), which was designed for spinning disks. In the era of flash, SCSI is the bottleneck. **NVMe-oF (NVMe over Fabrics)** allows the low-latency NVMe command set to run over Ethernet.
Latency Forensics: RDMA vs. TCP
While NVMe-oF can run over standard TCP, the highest performance is achieved using **RDMA (RoCE v2)**. This allows the storage controller to write data directly into the application's memory without touching the CPU.
Engineering Note: By using NVMe-oF over RDMA, we reduce the to near zero, making remote storage perform as if it were a local PCIe drive.
6. Optical Hydraulics: 400G, 800G & PAM4 Signaling
At 100G and above, we can no longer use simple NRZ (Non-Return-to-Zero) signaling. We use **PAM4 (Pulse Amplitude Modulation)**, which encodes 2 bits per clock cycle by using four distinct voltage levels.
Signal Integrity Forensics
PAM4 is highly sensitive to noise. We use **FEC (Forward Error Correction)** to reconstruct corrupted bits in real-time. If a 400G link shows high pre-FEC error rates, it's often a sign of a dirty fiber connector or a failing transceiver laser.
Co-Packaged Optics (CPO)
As we move to 1.6T and beyond, the distance between the switch chip and the transceiver is too long for copper traces. CPO moves the laser engines directly onto the same substrate as the silicon, eliminating the 'Copper Wall'.
7. NetDevOps: Orchestrating the Fabric
A modern data center has thousands of switches. Configuring them manually is impossible. We use **Terraform** and **Ansible** to treat the network as code.
GitOps & Drift Forensics
The 'Source of Truth' is the Git repository. If a manual change is made on a switch, the orchestration engine detects the 'Drift' and automatically reverts it.
Forensic Insight: Most network outages in 2026 are caused by 'Configuration Collision'—where two automated scripts try to update the same BGP policy simultaneously.
8. Disaster Recovery: The Speed of Light Constraint
You cannot defeat physics. The speed of light in fiber is ~200,000 km/s. This means for every 100km of distance, you add ~1ms of round-trip latency. This is the ultimate constraint for **Active-Active** data centers.
Synchronous vs. Asynchronous Replication
If your data centers are more than 50km apart, you cannot use synchronous replication without killing application performance. You must move to **Asynchronous** flows, which introduces the risk of **RPO (Recovery Point Objective)** data loss.
GSLB (Global Server Load Balancing)
Using DNS to steer users to the closest healthy data center based on health checks and RTT (Round Trip Time).
LISP (Locator/ID Separation Protocol)
Allowing an IP address to move between data centers without changing its 'Identity', enabling seamless VM migration across regions.
9. Data Center Economics: The Cost of a Port
Infrastructure engineering is ultimately about ROI. A 400G switch port might cost $5,000, but the **OpEx** (Power, Cooling, Maintenance) over 5 years is often 3x the **CapEx**.
The Blast Radius Math
We use smaller 'Fault Domains' to ensure that a single failure doesn't take out the whole cloud. The smaller the fault domain, the higher the cost per port, but the lower the risk of a global outage.
Founders Insight: "Building a data center that never fails is easy if you have infinite money. Building one that is 99.999% reliable for the lowest possible cost—that is engineering."
Frequently Asked Questions
Technical Standards & References
Related Engineering Resources
BGP-EVPN Control Plane: MP-BGP, Route Types, and the Modern Fabric Brain
The BGP-EVPN (Ethernet VPN) control plane is the central nervous system of the modern data center fabric, responsible for distributing reachability information for both Layer 2 MAC addresses and Layer 3 IP prefixes across the spine-and-leaf topology. Unlike traditional spanning-tree-based networks where MAC addresses are learned by flooding unknown unicast frames, EVPN uses BGP as a control plane to advertise MAC and IP reachability between leaf switches in a more efficient and deterministic manner. The EVPN address family, defined in RFC 7432, introduces five route types that serve different purposes in the fabric: Route Type 2 (MAC/IP Advertisement) is the most commonly used, carrying the MAC address, IP address, and VXLAN Network Identifier (VNI) for each host or virtual machine connected to the leaf switch. When a VM boots up on a server connected to Leaf-1, that leaf switch learns the VM's MAC and IP addresses (through local ARP or DHCP snooping) and generates an MP-BGP update containing a Route Type 2 advertisement, which is sent to all other leaf switches in the fabric via the Route Reflector (RR). The other leaf switches install the MAC and IP addresses in their forwarding tables, enabling direct VXLAN-encapsulated communication between any two VMs without requiring ARP flooding or unknown unicast flooding across the fabric.
The EVPN Route Reflector (RR) architecture is a critical design element that determines the scalability and convergence performance of the fabric. In a data center with 200 leaf switches, each leaf forms an MP-BGP session with the Route Reflectors (typically deployed as a pair of redundant servers or as a function within the spine switches). The RR receives EVPN routes from all leaf switches and reflects them to all other leaf switches, eliminating the need for a full mesh of BGP sessions between all leaf pairs (which would require 19,900 sessions for 200 leaf switches). The RR does not modify the EVPN routes it reflects; it simply passes them through, acting as a "route server" rather than a "route processor." The placement of the RR in the network is critical: the RR should be in the same Layer 3 domain as the leaf switches, with a service-level agreement (SLA) for BGP session convergence of less than 1 second in the event of an RR failure. A common deployment model is to run the RR function on a dedicated pair of servers running a BGP daemon (such as FRRouting or Bird), connected to the fabric via dual-homed connections to two different spine switches, ensuring that an RR failure or a spine switch failure does not disrupt the EVPN control plane.
The EVPN route resolution process is where the control plane interacts with the data plane to determine the optimal path for each traffic flow. When Leaf-1 receives an EVPN Route Type 2 advertisement for a MAC address from Leaf-2, it installs the MAC address in its Layer 2 forwarding table with a next-hop of Leaf-2's VTEP IP address (typically the loopback interface of Leaf-2). When a locally connected VM sends a packet to the advertised MAC address, Leaf-1 encapsulates the Ethernet frame in a VXLAN header with the outer destination IP set to Leaf-2's VTEP IP and forwards it through the IP fabric. If multiple leaf switches advertise the same MAC address (as in an anycast gateway deployment where multiple leaf switches serve the same IP subnet), the leaf switch selects the best path based on the BGP path selection algorithm: the route with the lowest IP prefix length, then the lowest origin code, then the lowest MED, and finally the lowest next-hop IP address. This deterministic path selection ensures that traffic from Leaf-1 to a multi-homed MAC address always takes the same path, preventing the MAC flapping and forwarding loops that would occur if the leaf switch used ECMP to distribute traffic to multiple next-hops for the same MAC address.
The operational management of the EVPN control plane requires careful monitoring of the route table size and the BGP session stability. In a large data center with 10,000+ tenant VMs, the EVPN route table can grow to 100,000+ routes (each VM is typically advertised as a Route Type 2 entry with both MAC and IP information). The Route Reflector must be sized to handle this route scale: a typical Route Reflector server with 32 GB of RAM can handle approximately 500,000 EVPN routes, but the BGP update processing rate (routes per second) is often the limiting factor rather than the memory capacity. During a leaf switch reboot, the RR must process the entire route table advertisement from the rebooting leaf (up to 10,000 routes for a leaf with 100 connected ToR switches, each serving 100 VMs), which can take 5-10 seconds depending on the BGP update pacing. During this window, the other leaf switches have stale forwarding information for the routes that were previously advertised by the rebooting leaf, potentially causing traffic to be black-holed or misrouted. The solution is to implement "graceful restart" or "non-stop routing" on the leaf switches, which retains the forwarding entries for the routes learned from the rebooting leaf until the BGP session is re-established and the routes are verified to be still valid.
The evolution of the EVPN control plane is moving toward a "unified" fabric where EVPN carries both Layer 2 MAC addresses and Layer 3 IP prefixes in a single address family, eliminating the need for separate OSPF or IS-IS underlay routing. In the unified EVPN approach, the underlay IP routing (the loopback interfaces and inter-switch links) is carried by a lightweight IGP such as OSPF or IS-IS with a very small database (only the leaf loopbacks and the spine loopbacks, typically fewer than 500 routes for a 200-leaf fabric). The tenant routes (MAC addresses and host IP prefixes) are carried by EVPN. The IP prefixes for anycast gateway services (the SVI IP addresses that serve as the default gateway for each tenant VLAN) are also carried by EVPN as Route Type 5 (IP Prefix) advertisements. This unified EVPN approach significantly simplifies the fabric configuration because all tenant routing information flows through a single BGP session, and it aligns with the industry trend toward "BGP-only" data center fabrics that rely on a single routing protocol for all control plane functions. The unified EVPN model is supported by all major data center switching platforms (Cisco NX-OS, Arista EOS, Juniper JunOS, NVIDIA Cumulus) and is expected to become the dominant data center fabric architecture over the next 3-5 years, replacing the more complex multi-protocol architectures that have been common in data center deployments since the introduction of VXLAN in 2014.
RDMA over Converged Ethernet: RoCEv2 Fabric Design for AI and HPC Workloads
The emergence of artificial intelligence (AI) and high-performance computing (HPC) workloads as the dominant data center traffic type has driven a fundamental rethinking of data center network design. Traditional TCP-based communication is inadequate for the distributed training of large language models (LLMs) and other AI workloads because TCP's kernel involvement, data copying, and context switching add tens of microseconds of latency per operation, which accumulates catastrophically across the thousands of parallel communication operations required for gradient synchronization in distributed training. Remote Direct Memory Access (RDMA) over Converged Ethernet version 2 (RoCEv2) addresses this by allowing one server's GPU to directly read from or write to another server's GPU memory over the Ethernet network without involving the CPU, operating system, or application buffers. The RoCEv2 protocol encapsulates InfiniBand-style RDMA operations over a UDP/IP transport (destination port 4791), enabling RDMA communication to traverse standard IP-routed Ethernet networks without requiring specialized InfiniBand hardware. The per-operation latency of RoCEv2 is typically 1-3 microseconds for a small message on a lightly loaded network, compared to 10-50 microseconds for TCP—a 10-50x improvement that is essential for scaling AI training across thousands of GPUs.
The deployment of RoCEv2 in a data center fabric requires a lossless transport network because RDMA operations assume that packets are never dropped. A single dropped RoCEv2 packet can stall the entire RDMA operation for hundreds of milliseconds (the timeout before the RDMA transport layer detects the loss and requests retransmission), which is catastrophic for AI training where thousands of GPUs are synchronizing gradients every few seconds. The lossless transport is achieved through Priority Flow Control (PFC, IEEE 802.1Qbb), a link-level flow control mechanism that operates on a per-priority basis. PFC allows a receiving switch to send a "pause" frame to the sending switch when its input buffer exceeds a threshold, instructing the sender to stop transmitting on that priority class for a specified duration. The PFC pause frames operate on the same 802.1p priority classes as the standard Ethernet QoS marking, allowing the network engineer to designate one or two priority classes as "lossless" (PFC-enabled) for RoCEv2 traffic while keeping the remaining priorities as "lossy" (standard tail-drop) for TCP and other traffic. The configuration of the PFC thresholds is critical: if the pause threshold is set too low, the link is frequently paused even under light load, reducing throughput; if it is set too high, the switch buffer overflows before the pause takes effect, causing drops.
The network-wide coordination of PFC is the most challenging aspect of RoCEv2 fabric design. In a multi-hop spine-and-leaf fabric, PFC must be configured consistently on every switch in the path between the two communicating servers. If one switch in the path does not support PFC or has PFC disabled, a burst of traffic from that switch can overflow the input buffer of the downstream switch, causing packet loss that stalls the RDMA operation. The industry has standardized on the Data Center Quantized Congestion Notification (DCQCN, RFC 6836) mechanism to provide end-to-end congestion control for RoCEv2 traffic, reducing the reliance on PFC for lossless transport. DCQCN works by having the receiver generate Congestion Notification Packets (CNPs) when it detects congestion (indicated by packets arriving with the Explicit Congestion Notification ECN flag set). The sender reacts to CNPs by reducing its transmission rate, similar to how TCP congestion control reduces the window size in response to packet loss. The combination of DCQCN (end-to-end rate control) and PFC (link-level flow control) provides a robust lossless transport that maintains high throughput even under congestion, but it requires careful tuning of the ECN marking thresholds, the CNP generation rate, and the rate reduction factor to achieve optimal performance.
The network topology for AI training clusters imposes specific requirements on the spine-and-leaf fabric design. AI training workloads exhibit an "all-to-all" communication pattern where every GPU must exchange gradients with every other GPU in the training job, creating a traffic matrix that is fundamentally different from the "east-west" traffic patterns of traditional data center applications. The traditional leaf-spine topology with 40-80 leaf switches and 4-8 spine switches provides adequate capacity for traditional applications but becomes a bottleneck for AI training because the spine-to-spine bandwidth is limited by the number of spine switches. The recommended topology for AI training clusters is a "rail-optimized" or "fat-tree" topology where the number of spine switches equals the number of leaf switches, providing full bisection bandwidth between any pair of leaf switches. In a 128-GPU training cluster (64 servers with 8 GPUs each, connected to 8 leaf switches with 8 spine switches), the rail-optimized topology provides 800 Gbps of bandwidth between any two leaf switches (with 8 spine switches each providing 100 Gbps), ensuring that the gradient synchronization step of the training process is not limited by the network bandwidth. The additional cost of the rail-optimized topology (a 2:1 spine-to-leaf ratio compared to the typical 4:1 or 8:1 ratio in traditional data centers) is justified by the 2-5x improvement in AI training throughput that it enables.
The operational management of a RoCEv2 fabric requires specialized monitoring tools that go beyond the standard SNMP-based network monitoring used in traditional data centers. The key performance indicators for a RoCEv2 fabric include: PFC pause frame counters (on all interfaces, per priority class), ECN marking percentage (on all interfaces, per priority class), CNP generation rate (per receiver), and the RDMA read/write completion rate (per GPU-to-GPU flow). The PFC pause frame counters are the most important early warning signal: an increase in pause frame count indicates that the ingress buffer on the downstream switch is building up, which precedes packet loss if the trend continues. The ECN marking percentage indicates the level of congestion in the fabric: a marking rate above 1% suggests that the fabric is approaching its capacity limit and may need to be expanded. The CNP generation rate indicates the effectiveness of the DCQCN congestion control: a high CNP rate with low throughput suggests that the DCQCN parameters are too aggressive and need to be relaxed. These RoCEv2-specific metrics are collected by the network monitoring system at 10-second intervals and are displayed on a dashboard that provides the network engineering team with real-time visibility into the health and performance of the AI training fabric. The integration of these monitoring capabilities with the cluster scheduler (such as NVIDIA's DGX A100 base command scheduler or the open-source SLURM workload manager) allows the scheduler to make intelligent placement decisions, ensuring that GPU-to-GPU communication paths stay within the same spine switch group to minimize latency and maximize training throughput.
"You are our partner in accuracy. If you spot a discrepancy in calculations, a technical typo, or have a field insight to share, don't hesitate to reach out. Your expertise helps us maintain the highest standards of reliability."
Contributors are acknowledged in our technical updates.