Parallelism: Choosing Your Bottleneck

Expert Parallelism (EP) — the sharding of MoE expert parameters across GPUs — creates a fundamentally different network traffic pattern than Data Parallelism or Tensor Parallelism. In EP, each GPU holds a subset of the expert weights, and tokens must be routed to the GPU hosting their assigned expert via an **All-to-All** collective. Unlike the ring-based All-Reduce of Data Parallelism where traffic patterns are predictable and uniform, All-to-All is a permutation-based pattern where every GPU sends a different amount of data to every other GPU. This asymmetry makes EP extremely sensitive to the network topology and load balancing scheme.

The key metric for EP network sensitivity is the **Permutation Matrix Sparsity** — the fraction of GPU pairs that must exchange data in a given training step. In a 64-GPU expert parallel group with 128 experts, each GPU hosts 2 experts and processes 64 tokens per step. Each token is routed to one of 128 experts, creating a 64 x 64 permutation matrix with an average density of 3.1% (each GPU sends to approximately 2 of 63 peers). The All-to-All collective must route these 64 tokens per GPU across the fabric, with each token representing a 2 KB activation vector. The total data volume per step is 64 GPUs x 64 tokens x 2 KB = 8 MB — trivial bandwidth-wise but latency-critical because the All-to-All is a barrier operation.

The latency sensitivity arises from the **Straggler Effect** in All-to-All. The collective completes only when the last GPU has received all its tokens. In a Dragonfly topology with 4 global links per group, a GPU that must send 10 tokens to a GPU in a different group competes for the limited global bandwidth. If the global link is congested with other All-to-All traffic, that GPU's token delivery is delayed, stalling the entire EP group. The straggler penalty in a 64-GPU EP group on a Dragonfly network can reach 50 microseconds over the ideal 5-microsecond All-to-All completion time — a 10x slowdown that directly increases the training step time.

The mitigation is **Topology-Aware Expert Placement** — placing experts on GPUs in the same network group to minimize cross-group traffic. DeepSpeed's EP scheduler assigns expert replicas such that 80% of token routing stays within the same leaf switch domain, reducing global link utilization by 75%. The scheduler uses a **Routing Affinity Matrix** built from the router's historical expert selection patterns to predict future routing and place experts accordingly. In production deployment on a 4,096-GPU cluster, topology-aware EP placement reduces the All-to-All completion time from 55 microseconds to 12 microseconds, improving the overall training throughput by 8% for MoE models with 128+ experts.

DeepSpeed's ZeRO (Zero Redundancy Optimizer) adds a fourth dimension to the parallelism matrix that is invisible in a simple DP/TP/PP view. In classic Data Parallelism, every GPU holds a full copy of the optimizer state (for Adam, that is 16 bytes per parameter: master weights, momentum, and variance). For a 175B-parameter model, each GPU wastes 2.8 TB of HBM just holding redundant optimizer state. ZeRO partitions that state across the DP ranks, so each GPU stores only its 1/N share. This triples the effective model size a cluster can train, but it introduces a new network traffic class: the optimizer-state shuffle at the start of every iteration.

The traffic pattern of ZeRO is fundamentally different from the All-Reduce of vanilla DDP. Instead of a global reduction where every rank exchanges with every other rank, ZeRO performs a **Reduce-Scatter followed by an All-Gather**. In stage 2, each GPU reduces its share of gradients (2M/N bytes for FP16 gradients), sends them to the rank that owns that partition, and then all ranks broadcast the freshly-updated optimizer state back. The volume of data moved is identical to All-Reduce — 2(N-1) x M/N per step — but the traffic is partitioned into non-overlapping flows, which changes how it contends on the fabric. A naive implementation sends the full parameter tensor from every rank to every other rank (O(N^2) total), which is why ZeRO requires a **bandwidth-optimal communication backend** — the same ring algorithm NCCL uses for All-Reduce.

The network design implication is that a ZeRO-2 or ZeRO-3 cluster does not need more bandwidth than a DP cluster, but it is far more sensitive to **link asymmetry** and **in-cast** congestion. Because each GPU's optimizer partition lives on exactly one rank, a single slow or oversubscribed link to that rank stalls the entire training step. Fabric designers for ZeRO workloads therefore favor topologies with fully-symmetric bisection bandwidth (Fat-Tree) over asymmetric designs, and they enable PFC (Priority Flow Control) to prevent the buffer-misery caused by many-to-one traffic bursts during the reduce-scatter phase.

Offloading further complicates the picture. ZeRO-Offload moves optimizer state to CPU DRAM or NVMe, removing the memory pressure but adding PCIe and network traffic: gradients must cross the PCIe bus to reach the CPU, and in multi-node configurations the optimizer updates are computed on the host that owns each partition. The observed effect is that communication and compute overlap windows change — the CPU becomes the latency bottleneck, and the network fabric sees smaller, more frequent transfers instead of one large burst. For architects, the takeaway is: always model the optimizer traffic class separately from the gradient All-Reduce class, because they have opposite burst profiles and different tolerance to loss.

The final design decision is not "which parallelism" but "where does each parallelism strategy live on the physical network." Every topology — NVLink domain, Fat-Tree, Dragonfly, or a hybrid — has a bandwidth and latency budget per link class, and the parallelism strategy must be mapped so that its most demanding communication stays on the fastest links. The standard mapping rule is: Tensor Parallelism stays inside the NVLink domain (because it exchanges tensors on every transformer layer), Pipeline Parallelism crosses NVLink domains but stays within the same rack or leaf group (because it exchanges activations once per micro-batch), and Data Parallelism spans the full fabric (because its All-Reduce is the only traffic that scales with total cluster size).

The mismatch cases are where the real costs appear. Placing TP-degree 8 across two NVLink domains (4 GPUs each) turns every layer's all-gather into a 50 GB/s (RoCE) exchange where the design assumed 900 GB/s (NVLink) — a 18x slowdown that makes the GPU sit idle through the critical path. Conversely, placing DP groups entirely within a single leaf domain while the global batch spans many leaves creates an **oversubscribed spine** during All-Reduce, because the traffic that should have crossed the fabric symmetrically now thunders through a handful of spine links. The fabric oversubscription ratio is the single most important number: a 3:1 oversubscribed spine will cap a DP-degree-512 All-Reduce at one-third of the theoretical bandwidth, and no amount of TCP or RoCE tuning recovers it.

Adaptive routing changes the calculus. Modern switches (e.g., NVIDIA Spectrum-4, Arista 7800R4) can hash flows dynamically across spine paths instead of static ECMP hashing. For All-Reduce ring traffic, which is bursty and many-to-many, adaptive routing reduces the tail latency by spreading the incast across all available links. However, adaptive routing is a double-edged sword for RDMA: out-of-order delivery breaks RoCEv2's expected-credit model unless the NIC supports reordering (which the ConnectX-7 and later do). The practical guidance is to enable adaptive routing only where the NIC and switch both support in-order delivery, and to benchmark the All-Reduce latency before and after — a 10% improvement in the collective phase is worth more than a 10% improvement in raw throughput, because the collective sits on the critical path of every training step.

Finally, remember that the topology decision is a co-design decision. A cluster built for TP-degree-8 with NVLink across all pairs cannot be retrofitted for TP-degree-16. A fabric bought with 1:1 oversubscription can absorb most parallelism strategy changes, but a 3:1 fabric silently caps the options you can ever run. When the choice is between buying more bandwidth and buying more GPUs, the bandwidth is almost always the correct long-term investment — because bandwidth is the only resource that every future parallelism strategy will demand more of.