Jodie Burchell
Dr. Jodie Burchell is the Team Lead of Python Advocacy at JetBrains, and was previously a Lead Data Scientist at Verve Group Europe. She completed a PhD in clinical psychology and a postdoc in biostatistics, before leaving academia for a data science career. She has worked for over 10 years as a data scientist in both Australia and Germany, developing a range of products including recommendation systems, analysis platforms, search engine improvements and audience profiling.
She has held a broad range of responsibilities in her career, doing everything from data analytics to maintaining machine learning solutions in production. She is a long time content creator in data science, across conference and user group presentations, books, webinars, and posts on both her own and JetBrain's blogs.
Session
Have you ever shipped a change because your eval score went from 76% to 85%, only to wonder afterwards what that number actually proved? Our evals are more sophisticated than ever: held-out test sets, calibrated LLM judges, statistically significant differences, and CI pipelines running automatically on every commit. So we can trust that the score really did change. But can we trust the conclusion we're drawing from it: that the system got better?
In this talk, we'll see that LLM evaluation is not just an engineering problem, but a measurement problem, and one that psychology has been grappling with for over a century. Psychometrics has spent decades asking how to measure things we can't observe directly, with the core concept of validity asking not just "is this score reliable?" but also "what evidence supports the interpretation I want to make from it?" We’ll start by seeing how benchmarks fail basic validity checks, including exploring this ourselves in a live demo. We’ll then bring the same perspective to the evals we build for real LLM applications, where metrics such as “helpfulness”, “quality” or “task success” can be just as difficult to interpret.
We'll close with four validity questions you can apply to your next eval, giving you a concrete checklist for thinking more carefully about what your metrics actually tell you. You'll leave with a practical way to distinguish between treating a score as a fact about your application and treating it as evidence for a claim about it, helping you build evals that tell you whether your changes are actually making the system better.