Ryan Marinelli
Ryan is a post-doctorial researcher at the University of Oslo with the Language Technology Group. He is working on Norwegian Language models and is an expert is benchmarking.
Session
Evaluating large language models is expensive. Standard benchmarking practices run fixed numbers of trials regardless of how quickly signal emerges, wasting compute when differences are obvious and under-sampling when they are subtle.
This talk introduces Bayesian sequential stopping rules as a principled alternative. Rather than committing to a sample size upfront, sequential methods continuously update a posterior over model performance and halt evaluation once sufficient evidence has accumulated, or once further sampling is unlikely to change the conclusion.
We walk through the statistical foundations, show how this framework applies to common benchmarking scenarios (accuracy comparisons, pass@k, agent task completion), and demonstrate a practical open-source Python implementation attendees can use immediately. Real benchmark datasets are used throughout to ground the approach.
Audience: ML engineers, researchers, and data scientists who run model evaluations and care about making reliable comparisons without burning unnecessary compute.
Takeaways: A conceptual understanding of sequential testing, practical guidance on when and how to apply it, and a working tool to take home.