Datacenter Fabrics

Cloud-Scale Fabric Engineering

Architecting the backbone of the cloud. From non-blocking Leaf-Spine topologies and EVPN control-planes to terabit-scale optical fabrics and liquid-cooled AI clusters.

Modern Datacenter Architecture

The modern datacenter has evolved from a simple server room into a highly engineered fabric designed to support cloud-scale workloads, AI training clusters, and global content delivery. At its core lies the network fabric — a carefully architected combination of spine and leaf switches, overlay tunnels, and distributed control planes that together deliver the non-blocking performance expected by modern applications.

Traditional three-tier designs (core-aggregation-access) have given way to leaf-spine Clos topologies that provide predictable latency, elastic scalability, and simplified troubleshooting. Every ToR (Top of Rack) leaf switch connects to every spine switch in a full mesh, ensuring that no packet traverses more than two switch hops regardless of the total fabric size. This architectural shift, combined with VXLAN overlay encapsulation and BGP-EVPN control planes, enables workloads to migrate freely across the fabric without IP address renumbering.

The rise of AI and machine learning workloads has further accelerated datacenter design innovation. GPU clusters require lossless RoCEv2 fabrics with PFC (Priority Flow Control) and ECN (Explicit Congestion Notification) to maintain training job throughput. A single LLM training run may involve thousands of GPUs spread across multiple racks, each exchanging gradients over the fabric at 400 Gbps or higher per link. The fabric design must guarantee that no single link failure can stall an entire training epoch.

This hub collects deep engineering resources across four critical domains: fabric topology design (leaf-spine, Clos, and non-blocking architectures), VXLAN overlay engineering (VTEP configuration, VNI design, and multi-tenancy), EVPN control plane (BGP-EVPN, MAC learning, anycast gateways), and datacenter operations (optics, cabling, power, and rack layout). Each resource includes theoretical foundations, production configuration examples, and forensic troubleshooting guides.

Engineering Guides

Core Domains

99.999%

Target Uptime

400G+

Per-Spine Link

Four Domains of Datacenter Fabric Engineering

Fabric & Topology

The foundation of any datacenter network is its physical topology. Leaf-spine Clos architectures provide predictable east-west latency by ensuring every leaf (ToR switch) connects to every spine in a full mesh. Key design parameters include oversubscription ratio (typically 3:1 to 4:1 for general-purpose fabrics, 1:1 for AI clusters), spine count based on radix limits, and link aggregation strategy. Modern 128-port 800G spine switches can support fabrics with over 8,000 100G server-facing ports at 4:1 oversubscription.

VXLAN Overlays

VXLAN (Virtual Extensible LAN) overlays decouple the virtual network from the physical underlay, enabling workloads to span racks, rows, and even data centers while maintaining Layer 2 adjacency. Each VXLAN Network Identifier (VNI) supports up to 16 million isolated segments, compared to the 4,096 VLAN limit of traditional 802.1Q. VTEP (VXLAN Tunnel Endpoint) configuration must account for underlay MTU (typically 9,000 bytes jumbo frames to accommodate the 50-byte VXLAN overhead), multicast or EVPN control plane, and ARP suppression.

EVPN Control Plane

BGP-EVPN (RFC 7432) provides a unified control plane for VXLAN overlays, replacing flooding-based learning with BGP-based distribution of MAC and IP routes. Type 2 routes advertise MAC/IP bindings, Type 3 routes handle IMET (Inclusive Multicast Ethernet Tag) groups, and Type 5 routes enable inter-subnet IP prefix advertisement. Anycast IP gateway (RFC 9136) allows every leaf to serve as the default gateway for any VLAN, eliminating the FHRP (First Hop Redundancy Protocol) bottleneck and enabling active-active gateway operation.

DC Operations

Beyond the control plane and topology design, successful datacenter operation requires meticulous attention to physical infrastructure. Optical transceiver selection (SR4, FR4, LR4, DR4, and ZR for different reach requirements), structured cabling with polarity management (methods A/B/C for MPO/MTP connectors), power budgeting per rack (typically 7-15 kW for standard racks, up to 40 kW for AI GPU racks), and cooling architecture (front-to-rear hot-aisle containment) all directly impact fabric reliability and performance.

Practical Example: EVPN-VXLAN Leaf Configuration

A typical EVPN-VXLAN leaf switch configuration (Cisco NX-OS style) for a spine-leaf fabric with anycast gateway:

! Spine-Leaf EVPN Configuration

feature nv overlay

feature bgp

nv overlay evpn

! Loopback for VTEP source and BGP peering

interface loopback0

ip address 10.1.1.1/32

! VXLAN Tunnel Interface

interface nve1

source-interface loopback0

member vni 10100

mcast-group 239.1.1.1

member vni 10200

mcast-group 239.1.2.1

! BGP EVPN Control Plane

router bgp 65001

neighbor 10.1.0.1 remote-as 65000

neighbor 10.1.0.2 remote-as 65000

address-family l2vpn evpn

send-community extended

vrf TENANT-A

address-family ipv4 unicast

! Anycast Gateway IP

ip virtual-router address 10.10.1.1/24

This configuration enables multi-tenant VXLAN overlays with BGP-EVPN route distribution across a spine-leaf fabric. Each leaf switch serves as an anycast gateway for its attached tenants, providing active-active first-hop redundancy.