Fat-Tree vs. Dragonfly: The Scaling Debate
Non-Blocking or Bust.
AI workloads exhibit a characteristic known as **All-to-All communication**. In a 32,000-GPU cluster, any single GPU may need to push data to any other GPU at full wire speed (400Gbps+).
If your network is "Oversubscribed," some GPUs will be throttled, leading to synchronization delays that can extend training time by weeks. This is why the **Non-Blocking Full Bisection Bandwidth (FBB)** metric is the holy grail of AI networking.
Fat-Tree (Clos)
The gold standard for AI. It uses 3 layers of switches (Leaf, Spine, Core) to ensure every path has the same hops and bandwidth. It is virtually immune to localized congestion.
Dragonfly
Connects switches in virtual groups. It reduces 'long' cables between tiers. While cheaper, it suffers from severe performance degradation if the traffic pattern becomes non-uniform.
Architectural Comparison Table
| Attribute | Fat-Tree (3-Tier) | Dragonfly |
|---|---|---|
| Max Scalability | ~128,000 Ports | ~1,000,000+ Ports |
| Optic Consumption | Extreme (N log N) | Optimized (O(1) groups) |
| Routing Strategy | Deterministic (LPR) | Highly Adaptive Required |
| Latency Jitter | Low (Uniform) | Moderate (Distance dependent) |
The Cabling Crisis.
Cabling density in a Fat-Tree is the primary operational hurdle. Every switch rack is a 'Spaghetti' of optical fiber. To manage this, hyperscalers are moving towards **LPO (Linear Drive Optics)** and **CPO (Co-Packaged Optics)** to reduce signal loss and power heat, even if the topology remains a Fat-Tree.
Calculated for a hypothetical 64K Cluster using Quantum-2 switches.