FP8 vs BF16 vs INT8: The Forensic Guide to AI Precision Scaling

Traditional scientific computing relies on **IEEE 754**, the standard for floating-point arithmetic. In this format, a number is represented by a sign bit, an exponent (determining the range), and a mantissa (the fractional part determining the precision).

For decades, **FP32** was the "Gold Standard." However, Deep Learning has a unique statistical signature: it is remarkably robust to noise but extremely sensitive to dynamic range. Gradient values during backpropagation often span multiple orders of magnitude, causing "underflow" if the exponent range is too narrow. This realization led to the divergence from scientific computing standards toward AI-specific formats like **BF16**.

Developed by Google Brain for the TPU v2, **BF16** was the first major "Deep Learning Native" format. It solved a critical problem with **FP16** (Half-Precision). FP16 uses 5 bits for the exponent, limiting its range to ~65,000. For LLMs, this often leads to gradient explosions that require complex "Loss Scaling" techniques.

With the NVIDIA H100 (Hopper), the industry transitioned to **FP8**. Unlike previous formats, FP8 is not a single standard but a dual-format system managed by a specialized hardware block: the **Transformer Engine**.

The "Magic" of FP8 lies in **Dynamic Scaling**. Because the range of FP8 is so small, you cannot just cast FP32 to FP8. The Transformer Engine monitors the distribution of values in every layer (the "Stats Collection") and calculates a scaling factor ($S$) that shifts the values into the representable range of FP8.

Forensic analysis of training runs often reveals failures not due to code bugs, but due to **Catastrophic Cancellation**. This occurs when two very similar numbers are subtracted, or when a very small number is added to a very large one.

To combat the precision loss of low-bit formats, high-end accelerators (like Graphcore IPU, Intel Gaudi, and now Blackwell) implement **Stochastic Rounding**.

Traditional rounding (round-to-nearest) always rounds 1.4 to 1.0. This introduces a persistent bias. Stochastic rounding treats the fractional part as a probability. If the value is 1.4, the hardware has a **40% chance** of rounding up to 2.0 and a **60% chance** of rounding down to 1.0. Over time, the expected value is exactly 1.4, statistically preserving the signal even when the hardware can't represent the digits.

With the NVIDIA Blackwell architecture, we are moving to **FP4**. Representing a number with only 4 bits (sign, 2 exponent, 1 mantissa, or similar) seems impossible. However, Blackwell introduces **MXFP4 (Microscaling Format)**.

By managing the dynamic range at the block level rather than the tensor level, Blackwell can achieve **20 PFLOPS of FP4 compute** in a single GPU, doubling the throughput of FP8 without the massive accuracy penalties of global quantization.

FlashAttention-3 is a critical software pillar for the FP8 era. Traditional attention mechanisms struggle with the "High Occupancy" requirement of Hopper and Blackwell. FlashAttention-3 introduces **Asynchronous Tiling**, allowing the Tensor Cores to perform FP8 matrix multiplications while the SM (Streaming Multiprocessor) simultaneously handles data movement from Shared Memory.

Moving data is expensive. Computing data is cheap. In a modern HBM3e-equipped GPU, moving 1 bit from the DRAM to the logic gates consumes roughly **100x more energy** than the actual floating-point operation.

This "Thermodynamic Tax" is the strongest argument for lower precision. By switching from FP32 to FP8, you aren't just saving memory capacity; you are reducing the total **Energy-per-Inference** by nearly 75%. In a 100,000-GPU cluster, this efficiency delta represents the difference between a 30MW facility and a 120MW facility—a saving of tens of millions of dollars in annual power costs.

Historically, **INT8** (8-bit Integer) was the king of inference. It was easy for CPUs and older GPUs to compute. However, INT8 is "Linear"—each step is the same size. Transformer weights and activations are "Logarithmic"—they have many values near zero and a few massive outliers.

**FP8** (Floating Point) is natively non-linear. The exponent structure allocates more "Resolution" to small values near zero, which is exactly where the majority of LLM weights live. This "Non-Linear Mapping" is why FP8 models consistently outperform INT8 models at the same bit-width, particularly as models get larger and the "Outlier problem" becomes more acute.

The **Chinchilla Scaling Laws** (DeepMind) suggest a specific ratio of compute ($C$) to parameters ($N$) and data ($D$). However, these laws assume 16-bit precision. Precision scaling introduces a new dimension: **Precision Hubris**.

When you reduce precision to 8-bit or 4-bit, you are effectively adding "Quantization Noise" to the model. To reach the same loss as a 16-bit model, you must either train a slightly larger model ($N$) or use more data ($D$). Hardware architecture in 2026 has decided that **Scaling N** via lower precision is significantly cheaper than **Scaling C** via higher precision. We are trading arithmetic accuracy for total knowledge capacity.

In long-context inference (1M+ tokens), the bottleneck is not the compute TFLOPS—it is the **KV Cache**. Every token generated must be stored in HBM to provide context for the next token.

Using **FP16** for the KV Cache is unsustainable; it consumes 2MB of memory per token for a 70B model. Modern inference engines (vLLM, TensorRT-LLM) now use **K-Cache Quantization**, shifting the KV store into **FP8** or even **INT4**. Forensic testing shows that since the KV Cache is effectively the model's "Short-Term Memory," it can tolerate lower precision much better than the model's "Long-Term Weight Storage," provided the scaling factors are updated per-head in the attention block.