Cluster
Sync.

In a global circuit, every 1,000 km adds approximately 10ms of RTT. For a Spanner-style globally synchronous database, this is the floor of your performance.

Even with perfect fiber, clocks are unreliable. In a multi-cluster setup, Cluster A's clock might be 50ms ahead of Cluster B.

• TrueTime & GPS

  Google Spanner uses specialized GPS clocks and atomic oscillators to provide a "Confidence Interval" for the current time. This allows the database to "Wait Out" the uncertainty, ensuring linearizability without a central bottleneck.

Conflict resolution is the defining design dimension of multi-cluster state synchronization systems. The three major approaches — Last-Writer-Wins (LWW), CRDT-based (Conflict-Free Replicated Data Type) automatic merge, and Operational Transformation (OT) — each impose fundamentally different bandwidth, latency, and state convergences characteristics. LWW (implemented by Kubernetes Federation v2's `kubefed2` and `karmada`) resolves conflicts by timestamp comparison: each write carries the wall-clock time from the originating cluster's NTP-synchronized clock, and the value with the most recent timestamp is selected by the resource aggregator. The simplicity of LWW comes at the cost of clock-skew-related inconsistencies: if cluster A's NTP offset drifts by +100ms relative to cluster B, and two operators simultaneously update the same configmap at the same logical time, the value from cluster A consistently wins even when cluster B's operation was causally later. With a typical NTP synchronization accuracy of ±5 ms in a WAN-connected multi-cluster environment (SL2775 stratum-2 servers with 10 ms RTT), the probability of an inversion error (wrong writer winning) is approximately P_inversion = P(|skew| > Δt_causal) / 2, where Δt_causal is the time between the two conflicting writes. For writes arriving within 10 ms (the NTP skew bound), P_inversion ≈ 50% — meaning LWW produces the correct causal result only by chance for near-simultaneous updates. The sync tool's conflict model computes P_inversion as a function of the NTP stratum level and the inter-cluster RTT, and it recommends LWW only for configurations where the expected write inter-arrival time exceeds 3× the NTP maximum skew.

CRDT-based conflict resolution (used by the `ocm` open-cluster-management project and the `kcp` Kubernetes-like API) eliminates the clock dependency by using mathematical merge rules that guarantee convergence regardless of the operation order. The most widely used CRDT for Kubernetes resources is the Observed-Remove Set (OR-Set): each replica maintains a set of (value, tag) pairs, where tag is a globally unique identifier (typically a (cluster_id, monotonic_counter) tuple). An add operation inserts (value, tag) into the set; a remove operation adds all tags observed by the remover to a tombstone set (the "observed" component). When two replicas merge, the result is the set of all (value, tag) pairs whose tags are not in the other replica's tombstones. The OR-Set guarantees no data loss (the merge algebra is monotonic) and eventual convergence to the same state at all replicas. The bandwidth cost of CRDT is the per-resource metadata overhead: each replica must transmit both the current value tags and the tombstone set, which grows linearly with the total number of delete operations over the resource's lifetime. For a namespace with 10,000 configmaps and a yearly churn of 100,000 create/delete cycles, the tombstone set reaches approximately 100,000 entries × 16 bytes per tag (8 bytes cluster_id + 8 bytes counter) = 1.6 MB per resource type per cluster — negligible for a 1 Gbps inter-cluster link (16 ms sync time) but problematic for low-bandwidth interconnects (10 Mbps, taking 1.3 seconds per full state sync). The tool's CRDT bandwidth model computes the tombstone set size as a function of the resource churn rate and the sync interval, and it recommends tombstone GC (garbage collection) thresholds where tombstones older than T_gc are explicitly acknowledged by all replicas and then pruned from the CRDT state.

Operational Transformation (OT) — the approach used by Google Docs and `yjs` (Y.js) for collaborative editing — resolves conflicts at the individual operation level rather than the state level. In the Kubernetes context, OT applies to resource spec patches: when clusters A and B both patch the same deployment's replica count, the OT merge algorithm transforms each operation against concurrent operations to produce a convergent result. The transformation function T(op_a, op_b) adjusts op_a to account for op_b's effect on the document state. For numeric fields (e.g., replicas), the transform is typically commutative-addition: op_a = "set replicas = replicas + 5" and op_b = "set replicas = set_replicas(replicas = 3)" — the OT engine transforms op_a to "set replicas = replicas + 0" (because op_b overwrites the replicas field), and the final state is replicas = 3. The OT algorithm requires that all transformations are invertible and that the operation history is available for transform — which means each cluster must maintain an ordered operation log that can grow unboundedly if the commit interval is long. The OT log size L_ot = r_write × T_sync, where r_write is the write rate on the resource and T_sync is the cross-cluster sync latency. For r_write = 1 write/second (a moderate Helm upgrade cadence) and T_sync = 100 ms, L_ot = 0.1 operations — effectively zero. But for r_write = 100 writes/second (a CI/CD pipeline continuously updating a canary deployment) and T_sync = 5 seconds (a typical federation controller sync interval), L_ot = 500 operations, requiring approximately 500 × 256 bytes per OT operation = 128 KB of operation log per resource — manageable for a few resources but unbounded for 10,000+ resources (1.28 GB total OT log). The sync tool's OT memory model warns operators when the aggregate OT operation log across all tracked resources exceeds 25% of the federation controller's available heap (default 512 MB for the `kubefed2` controller), at which point the controller should be scaled vertically or the number of conflict-tracked resources should be reduced.