Beyond LLM-as-Judge: Using GLIDE for Reliable, Scalable Evaluation of GenAI Systems
2026-09-12 –, Room A All times in Europe/Amsterdam
Agentic systems are booming, yet few reach production. A primary blocker is performance evaluation on unstructured outputs (typically text): LLM-as-Judge is cheap but biased while human annotations don't scale. To bridge this gap, practitioners need scalability without sacrificing statistical rigor.
This talk introduces Prediction-Powered Inference, a framework that combines a small set of human labels with a large volume of proxy judgments to produce unbiased performance measurements with formal confidence intervals. We will showcase GLIDE, a new Python open-source package that makes state-of-the-art techniques like PPI++ accessible to every developer.
Target Audience: ML and Software Engineers struggling to evaluate GenAI systems, and researchers in the field of prediction-powered inference. No background required.
Takeaways & Materials: All code and tutorials, will be shared via our open-source repository.
This session dives into rigorous evaluation methods, following prediction-powered statistical methods for AI system evaluation, as packaged in the GLIDE library.
The package allows engineers to leverage the seminal work of Prediction-Powered-Inference (PPI++) and Active Statistical Inference to correct for LLM-as-Judge bias. GLIDE provides statistical guarantees that are often missing in modern LLM evaluation pipelines.
The presentation will outline the methodology of evaluation-driven-development and showcase live demos using GLIDE on a concrete use case.
Outline
• Introduction (5 min): The Agentic Boom and the Reliability Crisis
• The Methodology (10 min): High-level intuition of prediction-powered inference
• GLIDE in Action (10 min): Live Demo, using GLIDE to evaluate a RAG agent
• Future & Community (5 min): The GLIDE roadmap and how to contribute
With a career rooted in the rigors of General Relativity, I specialized in building robust, ethical, and scalable AI systems. I have several years of experiences contributing to the open-source community, having led the development of MAPIE, the reference library in conformal predictions. I now work on reliable and scalable evaluation of GenAI systems, and took the lead of GLIDE, a new open-source package implementing state-of-the-art semi-supervised inference techniques.