BEGIN:VCALENDAR
VERSION:2.0
PRODID:-//pretalx//pretalx.com//pydata-amsterdam2026//talk//J9W3GE
BEGIN:VTIMEZONE
TZID:CET
BEGIN:STANDARD
DTSTART:20001029T040000
RRULE:FREQ=YEARLY;BYDAY=-1SU;BYMONTH=10
TZNAME:CET
TZOFFSETFROM:+0200
TZOFFSETTO:+0100
END:STANDARD
BEGIN:DAYLIGHT
DTSTART:20000326T030000
RRULE:FREQ=YEARLY;BYDAY=-1SU;BYMONTH=3
TZNAME:CEST
TZOFFSETFROM:+0100
TZOFFSETTO:+0200
END:DAYLIGHT
END:VTIMEZONE
BEGIN:VEVENT
UID:pretalx-pydata-amsterdam2026-J9W3GE@pretalx.com
DTSTART;TZID=CET:20260910T115500
DTEND;TZID=CET:20260910T122500
DESCRIPTION:LLM training demands ever more GPU-hours and memory\, making an
 y savings extremely valuable. As models grow\, BF16 is increasingly expens
 ive in both compute and memory. FP8 is rapidly becoming the standard for l
 arge-scale pretraining\, yet it remains a mystery to most practitioners du
 e to the numerical tricks it requires.\nThis talk demystifies FP8 training
 \, drawing from hands-on experience implementing FP8 in large-scale produc
 tion LLM pretraining: what happens under the hood of libraries like Transf
 ormer Engine and torchao\, why a naive approach breaks training\, and what
  practical approaches keep it stable — including scaling strategies\, fo
 rmat choices\, and selective higher-precision fallbacks. We also take a br
 ief look at advanced techniques pushing FP8 beyond matrix multiplications.
 \nYou will leave with a clear understanding of how FP8 training works\, wh
 at can go wrong\, and how to use it in your pipeline.\nAnyone interested i
 n modern LLM pretraining is welcome. This talk will be especially relevant
  for LLM practitioners and performance engineers.
DTSTAMP:20260710T141321Z
LOCATION:Room 2 (350)
SUMMARY:LLMs on a Diet: Low-bit pretraining at scale with FP8 - Arkadii
URL:https://pretalx.com/pydata-amsterdam2026/talk/J9W3GE/
END:VEVENT
END:VCALENDAR