BEGIN:VCALENDAR
VERSION:2.0
PRODID:-//pretalx//pretalx.com//pydata-amsterdam2026//speaker//AF3FUX
BEGIN:VTIMEZONE
TZID:Europe/Amsterdam
BEGIN:DAYLIGHT
DTSTART:20250912T000000
TZNAME:CEST
TZOFFSETFROM:+0200
TZOFFSETTO:+0200
END:DAYLIGHT
BEGIN:STANDARD
DTSTART:20251026T030000
RDATE:20261025T030000
TZNAME:CET
TZOFFSETFROM:+0200
TZOFFSETTO:+0100
END:STANDARD
BEGIN:DAYLIGHT
DTSTART:20260329T030000
RDATE:20270328T030000
TZNAME:CEST
TZOFFSETFROM:+0100
TZOFFSETTO:+0200
END:DAYLIGHT
END:VTIMEZONE
BEGIN:VEVENT
SUMMARY:Stop Early\, Decide Smarter: Bayesian Sequential Testing for LLM B
 enchmarking - Ryan Marinelli
DTSTART;TZID=Europe/Amsterdam:20260912T111000
DTEND;TZID=Europe/Amsterdam:20260912T121000
DTSTAMP:20260809T115056Z
UID:pretalx-pydata-amsterdam2026-BBXLSD@pretalx.com
DESCRIPTION:Evaluating large language models is expensive. Standard benchm
 arking practices run fixed numbers of trials regardless of how quickly sig
 nal emerges\, wasting compute when differences are obvious and under-sampl
 ing when they are subtle.\n\nThis talk introduces Bayesian sequential stop
 ping rules as a principled alternative. Rather than committing to a sample
  size upfront\, sequential methods continuously update a posterior over mo
 del performance and halt evaluation once sufficient evidence has accumulat
 ed\, or once further sampling is unlikely to change the conclusion.\n\nWe 
 walk through the statistical foundations\, show how this framework applies
  to common benchmarking scenarios (accuracy comparisons\, pass@k\, agent t
 ask completion)\, and demonstrate a practical open-source Python implement
 ation attendees can use immediately. Real benchmark datasets are used thro
 ughout to ground the approach.\n\nAudience: ML engineers\, researchers\, a
 nd data scientists who run model evaluations and care about making reliabl
 e comparisons without burning unnecessary compute.\n\nTakeaways: A concept
 ual understanding of sequential testing\, practical guidance on when and h
 ow to apply it\, and a working tool to take home.
LOCATION:Room A
URL:https://pretalx.com/pydata-amsterdam2026/talk/BBXLSD/
END:VEVENT
END:VCALENDAR