Global
Namespace.

In high-performance parallel file systems, streaming throughput (read/write) is often the headline spec, but **metadata operations** — file creates, deletes, renames, and stat calls — are the true bottleneck in AI training pipelines. A single epoch of a large-scale model may open, read metadata for, and close hundreds of thousands of small checkpoint shards. The latency of these metadata operations directly determines whether the GPU sits idle waiting for data.

Lustre addresses metadata through its **Distributed Namespace (DNE)** architecture, which scales metadata performance horizontally by partitioning the namespace across multiple **MDT (Metadata Target)** servers. Each MDT manages a disjoint set of directory entries. The **MDT Hash** function maps a file path to an MDT using a consistent hashing ring, ensuring that renames and directory moves preserve locality. The latest DNE2 update introduces **DirTree**, a B-tree structure stored on flash-backed NVMe, allowing each MDT to sustain 200,000+ metadata operations per second. A cluster with 16 MDTs can deliver 3.2 million metadata ops/sec, sufficient for the checkpoint workloads of a 10,000-GPU cluster.

BeeGFS takes a different approach, using a **Buddy Group Metadata** scheme. Instead of dedicated metadata servers, BeeGFS distributes metadata across the same nodes that serve data, assigning each directory a primary and secondary metadata buddy. The buddy pair maintains synchronous replication of the directory's inode table, ensuring that a single node failure does not orphan metadata. This co-location of metadata and data reduces the network hops required for a stat() call: the client can resolve a path in a single RTT by querying any node in the buddy group, rather than routing through a dedicated metadata server.

For AI checkpoint workloads, the practical difference manifests at scale. Lustre's dedicated MDTs provide deterministic metadata latency even under heavy data streaming, while BeeGFS's buddy system offers better aggregate throughput for mixed metadata+data workloads due to the elimination of the metadata network hop. Choosing between them requires modeling your specific metadata-to-data ratio: if your pipeline performs 100,000 file stat() calls per checkpoint, even a single millisecond of metadata latency adds 100 seconds of overhead to every epoch.

Lustre's Object Storage Targets (OSTs) are the physical units of storage that hold file data across the cluster. In a production AI deployment with 100+ OSTs distributed across 20 storage nodes, the OST utilization inevitably drifts over time due to the uneven distribution of file allocations. Some OSTs may reach 90% capacity while others sit at 30%, creating a hot-spotting problem where the most utilized OSTs become I/O bottlenecks for the entire training cluster. The process of redistributing data across OSTs to balance utilization — called **OST Rebalancing** — is one of the most disruptive operations in a Lustre filesystem.

The default Lustre stripe policy assigns files to OSTs using a round-robin allocation based on the file's **Object ID** hash. This provides statistically uniform distribution for a large number of small files. However, AI checkpoint files are large (100 GB to 7 TB) and relatively few in number (10-100 per training run). The round-robin allocation does not guarantee uniform distribution at this granularity — a single 7 TB checkpoint spread across 128 OSTs with 64 MB stripes creates 109,375 stripe objects, and the OST allocation may still deviate by 10-15% from uniform due to hash collisions and concurrent allocations.

Rebalancing involves migrating stripe objects from over-utilized OSTs to under-utilized OSTs while the filesystem remains online. The Lustre **lfs_migrate** tool performs this by creating a new version of the file with a different stripe map and deleting the old version after the copy completes. The migration throughput is limited by the aggregate network bandwidth between the OSTs — a factor often overlooked in capacity planning. In a 200 Gbps RoCE fabric connecting 20 OSTs, the inter-OST bandwidth is 400 Gbps per direction if all OSTs can simultaneously send and receive. At this rate, rebalancing a 10 TB set of data takes approximately 200 seconds. During this migration, the I/O bandwidth available to training workloads is reduced by the fraction consumed by the rebalancing traffic.

The **Progressive File Layout (PFL)** feature in Lustre 2.16+ provides a non-disruptive alternative. Instead of migrating existing data, PFL allows new files to use a different OST allocation policy without affecting existing files. An administrator can define a new **OST Pool** — a subset of OSTs with specific capacity and performance characteristics — and assign it to new checkpoint files while leaving old files on their original OSTs. The old files are migrated to the new pool during idle periods using a background daemon that respects I/O priority. This phased approach eliminates the throughput disruption of bulk rebalancing and allows the filesystem to accommodate evolving workload requirements without downtime. For AI training clusters where checkpoints follow predictable schedules, PFL-based rebalancing can be scheduled during the inter-job gap, completing the migration before the next training run begins.