PyData Amsterdam 2026

Using FP8 for pretraining is one of the most impactful yet under-explained optimizations available to LLM engineers today. This talk provides a practitioner-oriented walkthrough — from the "why" to the "how" — grounded in hands-on experience implementing FP8 in production-scale LLM pretraining. We'll go through what the core difference between FP8 and BF16 means for practitioners and show that this means we need additional tricks — a simple .to(fp8) won't work.

We will cover two main types of FP8 quantization (per-tensor and block-wise) and explain how they affect computations in LLM pretraining (forward and backward). We'll also cover the main pitfalls with FP8 training, such as using all operations in FP8 or applying it to lightweight operations where it hurts more than helps.

We will link our knowledge to what's happening under the hood in popular libraries such as Transformer Engine, torchao, and others. Next, we'll look at real production tricks to make it work even better — such as FP8 communications, quantization fusions, and rounding techniques.

This is a conceptual talk with code examples on slides. This talk focuses on FP8 for pretraining. We do not cover post-training quantization for inference or quantization-aware training.

Prior knowledge expected: Attendees should be familiar with the Transformer architecture and PyTorch. A basic understanding of data types is expected. Examples will use modern LLM architectures. No prior deep expertise in CUDA or quantization is required, though it is helpful.

Outline (30 min):

00:00–05:00 | Why FP8, why now. Modern accelerators offer nearly 2× higher throughput for FP8 operations compared to BF16, yet most training pipelines don't take advantage of this. We frame the problem in terms of compute efficiency and cost, and explain why FP8 for training is fundamentally different from post-training quantization for inference and how hardware opens these savings for us.

05:00–15:00 | FP8 under the hood. The anatomy of FP8 formats: E4M3 for forward activations and weights, E5M2 for gradients, and why this split matters. We then walk through why naive per-tensor quantization breaks training — activation outliers, loss of dynamic range — and how the field moved to more robust approaches: delayed scaling, block-wise (tile-wise) quantization, and selective higher-precision fallbacks for sensitive operations like embeddings and normalization layers.

15:00–25:00 | Making it work in practice. We'll go through the basic FP8 building block — the linear layer — and link our observations to how Transformer Engine, torchao, and other libraries abstract FP8 complexity for practitioners. We'll also show that existing open-source components can be composed into a flexible FP8 training setup tailored to your specific needs.

25:00–30:00 | Beyond GEMMs and wrap-up. A brief look at techniques pushing FP8 beyond matrix multiplications into communication-bound operations, and a teaser of what's coming next with newer hardware and formats. Q&A.

Target audience: ML engineers and LLM practitioners who train or fine-tune large models on GPUs. Also relevant for performance engineers interested in numerical optimization for training.

Audience takeaways:

• A clear understanding of why FP8 matters — both economically and technically
• Knowledge of what breaks when you quantize training (outliers, dynamic range, sensitive layers) and how modern scaling strategies fix it
• Practical familiarity with how Transformer Engine, torchao, and other libraries implement FP8 under the hood
• Awareness of production techniques pushing FP8 beyond matrix multiplications into communication and memory-bound operations

The talk is centered around the PyTorch ecosystem (torchao, Transformer Engine) and addresses a practical optimization that any engineer training large models on GPUs can benefit from. All discussed libraries are open-source.