LLM Evaluation in Production: A/B Testing and Observability
2026-09-12 –, Room B All times in Europe/AmsterdamEven small changes in LLMs can affect output quality, safety, and user experience. In this talk, we will show how to use Langfuse to log LLM outputs, run A/B tests, and track important metrics.
You will learn how to use data and experiments to improve your LLM applications and make better decisions during development.
Modern LLM development is not just about running a single model. Even small changes can affect output quality, safety, and user experience. In this session, attendees will learn how to use Langfuse to log LLM outputs, run A/B tests, and track important metrics.
We will also explore why A/B testing is important in LLM development and how to compare different models or prompt versions using real user data.
By the end of the session, attendees will understand how to:
Design data-driven A/B tests in LLM development workflows
Combine code, tests, and experiments to create reliable, repeatable testing processes
Log experiments and track model performance and user-facing outputs with Langfuse
Define and use simple metrics to evaluate LLM quality and behavior
Make better decisions by comparing different prompts or model versions
This session gives Python developers a framework to integrate experiments, tests, and data into LLM development, enabling reliable, repeatable, and data-driven workflows.
I have been a backend developer for 4 years, working primarily with Python and Django. I enjoy sharing what I’ve learned at previous PyCon talks and through writing on Medium, helping others improve their coding and AI skills.
Hello World, I'm Özge Çinko! 👋 I'm a computer engineer who finds inspiration at the intersection of curiosity and technology. Currently building the future as an AI Engineer at ING. For me, engineering is a creative craft - turning data into narratives and emotions into visual experiences. I am passionate about making technology more human-centric and purposeful. When I'm not coding, I'm usually writing, traveling, or chasing the thrill of learning something new.