Pipeline Parallelism (PP) distributes "chunks" of layers across nodes. While this saves memory, it introduces a sequential bottleneck. Node 1 must finish stage 1 before Node 2 can start. This leads to **Bubble Time** (Pipeline Bubbles), where GPUs sit idle.

While TP and PP split the model, Data Parallelism (DP) replicates the model and splits the training data. Microsoft's **ZeRO (Zero Redundancy Optimizer)** family changed the game by partitioning the optimizer state (ZeRO-1), gradients (ZeRO-2), and parameters (ZeRO-3).

To handle the 3D Parallelism traffic, modern data centers use a **Rail-Optimized** topology. In this design, GPU #1 from every node is connected to the same "Rail-1" switch.

• Inter-GPU Rails

  Ensures that All-Reduce operations for Data Parallelism happen within a single switch layer, minimizing "East-West" hops and congestion.

• Blocking vs. Non-Blocking

  For TP/PP clusters, anything less than a 1:1 non-blocking bisection bandwidth is unacceptable. A 3:1 oversubscription ratio can lead to a 50% drop in training FLOPS due to All-Reduce contention.

Pipeline model parallelism (GPipe, PipeDream, and their variants) partitions the model layers across N microbatches, and the total training step time is T_step = (T_f + T_b) × (N_micro − 1) / N_micro + T_f + T_b + T_comm, where T_f is the forward pass time per microbatch, T_b is the backward pass time per microbatch, and T_comm is the communication time for gradient synchronization. The pipeline bubble — the idle time at the beginning and end of each training step when not all microbatches are in flight — is a fundamental inefficiency proportional to (N_micro − 1) / (N_micro × (T_f + T_b)). For N_micro = 4 (the GPipe default), the bubble overhead is 3/4 = 75% of the per-microbatch time, reducing throughput by 42% compared to the ideal zero-bubble pipeline. Increasing N_micro to 32 reduces the bubble overhead to 31/32 ≈ 3.1% of the per-microbatch time, but the activation memory required for pipeline scheduling grows linearly with N_micro (each in-flight microbatch stores its activations for the backward pass). The memory cost per device is M_act = N_micro × Σ(layer) A_layer, where A_layer is the activation tensor size per layer. For a 175B-parameter model with FP16 activations and a sequence length of 2,048 tokens, each transformer layer's activations require approximately 4 MB per microbatch (key-value cache + hidden states + attention probabilities). With N_layers = 96 and N_micro = 32, M_act = 32 × 96 × 4 MB = 12.3 GB per device — consuming 38% of the A100-80GB HBM capacity before considering the model weights (approximately 350 GB across the partition) and the optimizer states (additional 700 GB across the partition for Adam with FP32 momentum and variance).

Communication-computation overlap is achieved by pipelining the all-reduce operation with the backward pass computation. The standard overlap schedule partitions the gradient tensor into N_chunks and initiates the all-reduce for chunk i while computing the backward pass for layer L − i (starting from the last layer). The overlap efficiency η_overlap = min(1, T_comm / (T_b + T_overhead)), where T_overhead is the additional synchronization cost for splitting the all-reduce into multiple smaller operations. With NCCL's all-reduce, splitting the gradient into 8 chunks increases the total communication time by approximately 10% due to additional kernel launch overhead (approximately 5 μs per chunk) and the suboptimal bandwidth utilization of small-message all-reduce (NVLink bandwidth utilization drops from 90% for a 1 GB all-reduce to 60% for a 128 MB all-reduce). The optimal chunk count N_chunk is therefore a trade-off: more chunks increase overlap potential but decrease communication efficiency. The analytical optimum occurs when d(η_overlap)/d(N_chunk) = 0, which for NCCL on an 8-GPU NVLink domain with gradient size G = 500 MB and T_b = 200 ms yields N_chunk_opt = 16 (T_comm_chunk = 31.25 MB × 2 passes / 300 GB/s ≈ 0.21 ms per chunk, T_overhead ≈ 5 μs per chunk, total T_comm = 16 × 0.215 ms = 3.44 ms). With N_chunk_opt = 16, the overlap fraction is T_comm / T_b = 3.44 / 200 = 1.72%, nearly perfectly hidden. The model parallelism bandwidth tool computes N_chunk_opt for the user's specific gradient size and NVLink topology, outputting the expected achieved throughput as T_step_achieved = T_step − η_overlap × T_comm.

Sequence parallelism (Megatron-SP, Ring Attention) introduces a different overlap opportunity: overlapping the all-reduce of the sequence-parallel dimension with the forward and backward computation of the tensor-parallel and pipeline-parallel dimensions. In a 3D parallel configuration with tensor parallelism across 8 GPUs (T=8), pipeline parallelism across 4 stages (P=4), and sequence parallelism across 2 GPUs (S=2), the total GPU count is T × P × S = 8 × 4 × 2 = 64. The sequence-parallel all-gather and reduce-scatter operations on the KV cache communicate 2 × S × H × bytes_per_element per attention layer over the sequence-parallel group (2 GPUs), which at 100 GB/s NVLink bandwidth takes approximately (2 × 2048 × 8192 × 2) / 100e9 = 0.67 μs per layer — negligible compared to the 3-5 ms per layer attention computation. However, for ultra-long sequences (128K tokens), the sequence-parallel communication grows to (2 × 131072 × 8192 × 2) / 100e9 = 43 μs per layer — still small but now 1-2% of the attention time, and the overlap is achieved by double-buffering the KV cache transfer: while one attention head computes on the local KV block, the DMA engine prefetches the next remote block over NVLink. This double-buffered overlap eliminates all sequence-parallel communication overhead for sequences up to 256K tokens at FP8 precision, making sequence parallelism essentially free for all practical training configurations. The tool models this by showing the sequence-parallel communication time as a separate bar in the training step breakdown, with and without double-buffer overlap, enabling operators to verify that their sequence length and parallelism degree fall within the free-overlap regime.