Reliable, rigorous, wrong: A psychometric view of LLM evals
Have you ever shipped a change because your eval score went from 76% to 85%, only to wonder afterwards what that number actually proved? Our evals are more sophisticated than ever: held-out test sets, calibrated LLM judges, statistically significant differences, and CI pipelines running automatically on every commit. So we can trust that the score really did change. But can we trust the conclusion we're drawing from it: that the system got better?
In this talk, we'll see that LLM evaluation is not just an engineering problem, but a measurement problem, and one that psychology has been grappling with for over a century. Psychometrics has spent decades asking how to measure things we can't observe directly, with the core concept of validity asking not just "is this score reliable?" but also "what evidence supports the interpretation I want to make from it?" We’ll start by seeing how benchmarks fail basic validity checks, including exploring this ourselves in a live demo. We’ll then bring the same perspective to the evals we build for real LLM applications, where metrics such as “helpfulness”, “quality” or “task success” can be just as difficult to interpret.
We'll close with four validity questions you can apply to your next eval, giving you a concrete checklist for thinking more carefully about what your metrics actually tell you. You'll leave with a practical way to distinguish between treating a score as a fact about your application and treating it as evidence for a claim about it, helping you build evals that tell you whether your changes are actually making the system better.
Modern LLM evaluation increasingly focuses on making scores more trustworthy: using held-out datasets, calibrating LLM judges, running evals in CI, controlling experimental variation, and applying rigorous statistics. These are all valuable techniques, but they leave a more fundamental question unanswered: what does the resulting score actually measure, and what conclusions does it justify?
This talk introduces that question through psychometrics, the field concerned with measuring constructs that cannot be observed directly. We’ll focus on the concept of validity: the evidence that supports the interpretation we want to make from a score. Applied to LLM evals, validity gives us a practical way to work backwards from the claim we care about. If we say an agent is “helpful”, for example, we can ask whether our eval cases actually sample helpful behaviour, whether irrelevant factors such as task structure change the score, whether independent measures of helpfulness agree, and whether better eval scores correspond to better outcomes in the real application.
We’ll make these ideas concrete using failures of LLM reasoning assessments. We’ll look at cases validity checks fail, and show that the supposed reasoning ability of LLMs are reflective of something rather different. A live Python demo will let us explore one of these effects ourselves. These examples provide controlled demonstrations of the same validity problems that can occur in application-specific evals.
We’ll then return to real LLM applications and show how validity can improve the way we design and interpret our evals. Rather than asking only “what metric should I track?”, we’ll start with “what claim do I want this eval to support?” and use that to guide the choice of cases, scoring criteria, robustness checks, corroborating evidence, and the boundaries within which the result should generalise.
The talk will cover:
- Why knowing that an eval score changed is different from knowing that a system improved.
- How psychometrics deals with latent constructs, and how validity turns an eval score into an evidence-based claim about a system.
- How to look for validity evidence in practice: whether an eval captures the intended behaviour, survives irrelevant changes, agrees with other measures, and predicts outcomes we care about.
- Examples from LLM reasoning assessments, including a live demonstration of how assessment conditions can change apparent model performance.
- Four questions to apply when designing your own evals: What claim am I trying to support? What else could move the score? What other evidence should agree with it? Where does the interpretation stop?
This is an intermediate talk but designed to be accessible to anyone who has worked with machine-learning metrics or LLM applications. No prior knowledge of psychometrics is required, and the core ideas will be introduced from first principles.
Dr. Jodie Burchell is the Team Lead of Python Advocacy at JetBrains, and was previously a Lead Data Scientist at Verve Group Europe. She completed a PhD in clinical psychology and a postdoc in biostatistics, before leaving academia for a data science career. She has worked for over 10 years as a data scientist in both Australia and Germany, developing a range of products including recommendation systems, analysis platforms, search engine improvements and audience profiling.
She has held a broad range of responsibilities in her career, doing everything from data analytics to maintaining machine learning solutions in production. She is a long time content creator in data science, across conference and user group presentations, books, webinars, and posts on both her own and JetBrain's blogs.