Scaling
Laws.

In the early days of deep learning, scaling was an exercise in trial and error. Today, it is a branch of predictive physics. Scaling Laws are empirical relationships that describe how the test loss of a Transformer model decreases as you increase three variables: the number of parameters (N), the amount of training data (D), and the total compute used for training (C).

The power of these laws lies in their predictability. By training a series of small "pilot" models (e.g., from 10M to 1B parameters), engineers can extrapolate exactly how a 175B or 1T parameter model will perform before investing tens of millions of dollars in GPU time. This predictability is what allows the AI industry to sustain its relentless pace of advancement. Without these laws, the risk of a $\$100\text{M}$ training run failing to meet performance targets would be too great for even the largest tech giants.

We observe that scaling is not merely about "bigger is better," but about the harmony of resources. If you scale parameters without scaling data, the model becomes a "memorizer" with poor generalization. If you scale data without parameters, the model hits a "capacity ceiling" where it cannot absorb new information regardless of how many tokens it sees. The scaling laws provide the exact ratio for this harmony.

The relationship between compute, data, and parameters is not merely linear; it follows a power-law distribution. The generalized loss function $L(N, D)$ can be expressed as:

Kaplan's original findings (2020) suggested that $\alpha \approx 0.05$ and $\beta \approx 0.06$. This indicated that increasing model size (N) was significantly more effective at reducing loss than increasing data (D). However, this "Model-First" philosophy led to models like GPT-3 being severely under-trained, as they possessed more "capacity" than they had "experience" (tokens).

When calculating the compute cost (C) of training a transformer, engineers use the 6ND rule of thumb. This approximation states that the total number of floating-point operations (FLOPs) required to train a model is roughly 6 times the product of the number of parameters (N) and the number of training tokens (D).

Where does the 6 come from? It is the sum of the forward and backward passes. For each token, the forward pass requires approximately 2N operations (one multiply-add per parameter). The backward pass is twice as expensive as the forward pass (4N ops), as it requires calculating both the gradients with respect to the activations and the gradients with respect to the weights.

This 6ND rule assumes no overhead from activation recomputation (gradient checkpointing). In reality, to save VRAM on massive models, engineers often recompute the forward pass during the backward pass. This increases the constant from 6 to 8 (or $8ND$). This tradeoff between compute and memory is a central decision in large-scale cluster orchestration.

Pure scaling laws assume "perfect" hardware where every FLOP is useful. In reality, scaling is bounded by the Model FLOP Utilization (MFU). As models scale, the ratio of communication (inter-node latency) to computation (intra-GPU math) increases, often causing the "observed" scaling law to flatten prematurely.

Most scaling laws assume training in FP16 or BF16. However, the industry is rapidly transitioning to FP8 and even INT4 for massive training runs. Precision scaling describes how reducing the number of bits per parameter affects the scaling coefficient.

Experimental data shows that while lower precision introduces "quantization noise," it allows for a larger number of parameters (N) to fit within the same memory and compute budget. For models beyond 500B parameters, the efficiency gains of FP8 often outweigh the slight increase in loss, effectively "stretching" the scaling law along the compute axis.

Chinchilla optimality assumes a constant data quality ($Q$). However, not all tokens are created equal. High-quality tokens (textbooks, verified code, logic puzzles) have a significantly higher Loss-Reduction Efficiency than low-quality web scrapes.

By filtering for quality, researchers can effectively shift the scaling curve downward. Training on a "Pure" dataset of 100B tokens can sometimes result in the same test loss as training on 500B tokens of "Noisy" data.

The order in which data is presented matters—a concept known as Curriculum Learning. Modern scaling recipes often involve "Data Annealing," where the model is trained on a massive, diverse dataset for 95% of the run, followed by a high-intensity "cool down" phase on ultra-high-quality data.

This final stage effectively "resolves" the model's knowledge, collapsing the final 5% of loss much faster than the initial power law would predict. Analyzing the scaling laws during the annealing phase reveals a "Breakthrough Regime" where intelligence gains become super-linear for a short period as the model synthesizes its latent representations.

The scaling laws for Mixture of Experts (MoE) follow a different trajectory than dense models. In an MoE architecture, the model consists of total parameters ($N_{total}$), but only a fraction are active ($N_{active}$) for any single token.

While dense models follow a strict 6ND compute cost, MoE models allow for Knowledge Scaling without Compute Scaling. By increasing the total number of experts, we can increase the model's "capacity" (memory) without increasing the FLOPs required for inference.

However, this introduces a Routing Penalty where the scaling efficiency drops slightly as the expert count increases due to load-balancing overhead and communication latency across GPU nodes (the "All-to-All" bottleneck).

As models grow beyond the memory of a single GPU, we must use 3D Parallelism (Data, Tensor, and Pipeline). Each of these introduces a "scaling tax" that bends the 6ND compute curve.

What happens when a model's optimal data requirement exceeds the total volume of high-quality human text on the internet? This is the Data Ceiling. In a data-constrained regime, scaling laws begin to "bend."

Recent research suggests that training for multiple epochs (up to 4-10 passes) over the same dataset can still yield improvements, though with diminishing returns compared to fresh tokens. The scaling coefficient ($\alpha$) effectively decays, necessitating a move toward Multi-Modal Data Scaling (training on video/audio) and Synthetic Data Augmentation.

The next frontier of scaling is not in training, but in Inference. Traditional transformers are "System 1" thinkers—they produce tokens in a single forward pass with a fixed compute-per-token cost. Inference Scaling allows a model to "think harder" by using more compute at the moment of generation.

This paradigm shift, popularised by models like OpenAI o1, introduces a new scaling law where accuracy scales as a function of Test-Time Compute. This involves techniques such as:

Eventually, scaling hits a physical wall: Power Density. As we scale clusters toward the "Gigawatt Era," the cost of cooling and power delivery begins to exceed the cost of the GPUs themselves.

Future scaling laws will likely need to incorporate Energy Efficiency (Tokens per Joule) as a primary variable. We are moving from a regime of "Intelligence at any cost" to one of "Intelligence within a fixed energy envelope," driving the development of specialized optical interconnects and advanced liquid-cooling manifolds to maintain the power-law trajectory.

Scaling laws are usually smooth power curves. However, certain capabilities—like mathematical reasoning or symbolic logic—exhibit Grokking, a phenomenon where a model's performance on a specific task suddenly jumps from 0% to nearly 100% after a critical training duration.

This suggests that the model is initially memorizing samples but eventually "discovers" the underlying algorithmic structure. Scaling the compute beyond the point of standard "convergence" can sometimes trigger these latent phase transitions, turning a mediocre model into a specialist.

Scaling Vision Transformers (ViT) and Video models (Sora/Lumina) follows a similar power law, but with different exponents. For vision, the scaling is often bound by Patch Size and Resolution.

Research into VLMs (Vision-Language Models) shows that the optimal parameter allocation shifting toward a 1:1 ratio between vision encoders and language decoders for high-reasoning tasks. However, video scaling is uniquely constrained by Temporal Resolution, where the compute cost scales cubically with the length of the video sequence if naive self-attention is used, necessitating the shift toward Diffusion Transformers (DiT).

At its core, scaling laws treat intelligence as a compression problem. If a model can predict the next token with high accuracy, it has successfully "compressed" the knowledge of the training set. The more bits of information we feed the model (D) and the more bits of capacity it has (N), the more intelligence we can extract.

This lead to the "Scale is All You Need" philosophy, which argues that we don't need "better" algorithms, just bigger ones. While true for foundational capabilities, the industry is now moving toward Context Scaling (long-context windows) and Action Scaling (training on real-world trajectories) to push the frontier further.

Can safety be scaled? Research into RLHF (Reinforcement Learning from Human Feedback) suggests that model "helpfulness" and "harmlessness" also follow predictable scaling patterns. However, alignment scaling often encounters a Safety-Utility Tradeoff, where pushing the model too hard toward strict alignment can cause a slight decay in its raw reasoning capabilities (the "Alignment Tax").

The future of scaling laws involves Constitutional AI, where a small set of human principles is used to supervise the scaling of a much larger child model, ensuring that as compute grows, the model's value alignment remains robustly locked to human intent.