Stop Early, Decide Smarter: Bayesian Sequential Testing for LLM Benchmarking
2026-09-12 –, Room A All times in Europe/AmsterdamEvaluating large language models is expensive. Standard benchmarking practices run fixed numbers of trials regardless of how quickly signal emerges, wasting compute when differences are obvious and under-sampling when they are subtle.
This talk introduces Bayesian sequential stopping rules as a principled alternative. Rather than committing to a sample size upfront, sequential methods continuously update a posterior over model performance and halt evaluation once sufficient evidence has accumulated, or once further sampling is unlikely to change the conclusion.
We walk through the statistical foundations, show how this framework applies to common benchmarking scenarios (accuracy comparisons, pass@k, agent task completion), and demonstrate a practical open-source Python implementation attendees can use immediately. Real benchmark datasets are used throughout to ground the approach.
Audience: ML engineers, researchers, and data scientists who run model evaluations and care about making reliable comparisons without burning unnecessary compute.
Takeaways: A conceptual understanding of sequential testing, practical guidance on when and how to apply it, and a working tool to take home.
Fixed-sample benchmarking is the default in NLP and LLM evaluation, but it has two failure modes: it over-runs when models are clearly different early on, and under-runs when tight comparisons require more evidence than the budget allows. Sequential hypothesis testing, long standard in clinical trials and A/B testing, offers a cleaner framework: keep sampling until a decision threshold is crossed, or until a maximum budget is exhausted.
This talk applies that framework specifically to LLM and agent benchmarking and covers:
The statistical case: Why pass@k evaluation is often wasteful, and what sequential Bayes factors and posterior stopping criteria offer instead
Design choices: Prior elicitation, stopping thresholds, and how to handle multi-model comparisons without inflating error rates
Practical implementation: A walkthrough of a pip-installable Python package built on this framework
Limitations and scope: When sequential testing helps, when it doesn't, and how to sanity-check results
The talk is self-contained; attendees need only basic familiarity with probability and Python.
Outline (30 min):
Motivation: the cost of fixed-N benchmarking — 5 min
Sequential testing foundations (Bayes factors, stopping rules) — 8 min
Demo: the Python package on real benchmark data — 10 min
Design guidance, edge cases, and Q&A — 7 min
Ryan is a post-doctorial researcher at the University of Oslo with the Language Technology Group. He is working on Norwegian Language models and is an expert is benchmarking.