Do you know how well your model is doing? Evaluate your LLMs
Large Language Models (LLMs) are becoming central to modern applications, yet effectively
evaluating their performance remains a significant challenge. How do you objectively compare different models, benchmark the impact of fine-tuning, or ensure your LLM responses adhere to safety guidelines (guard-railing)?
This hands-on workshop addresses these critical questions.
We will begin with an essential revision of the Hugging Face Transformers library, covering
basic LLM inference and fine-tuning. The core of the workshop will introduce and provide deep practice with Lighteval, an efficient and powerful LLM evaluation framework. Participants will learn how to leverage Lighteval to compare various LLMs available on the Hugging Face Hub using a range of pre-built tasks and metrics.
Finally, we will delve into advanced evaluation techniques, focusing on creating custom tasks and metrics tailored to unique, real-world application requirements. Participants will learn how to prepare custom datasets on the Hugging Face Hub and integrate them into Lighteval for precise, domain-specific evaluation. By the end of this workshop, you will possess the practical skills to rigorously evaluate, benchmark, and fine-tune your LLMs with confidence.
Outline:
Part 1
- Presentation: The importance of evaluation of LLMs
- Compare performance of LLM for specific tasks
- Benchmark the fine-tuning performance
- Rail guard the LLM responses
- Coding exercise: Introduction and revision of Hugging Face Transformers
- Revision of using Transformers for LLM influence
- Fine tuning a LLM with transformers
Part 2
- Presentation: Introduction of Lighteval
- What is Lighteval and what can it do
- Different tasks and metrics available in Lighteval
- Coding exercise: Using Lighteval to compare LLMs
- Familiar the use of Lighteval
- Compare two LLMs on Hugging Face Hub
- Experiment with different tasks and metrics
Part 3
- Presentation: Advance use of Lighteval
- Introduction of custom tasks and metrics
- What is needed for creating custom tasks and metrics
- How to put custom tasks and metrics together
- Coding exercise: Practice with custom tasks and metrics
- Uploading datasets to Hugging Face Hub
- Creating custom tasks and metrics
- Using custom tasks and metrics to compare LLMs
This workshop involves comparing different models, benchmarks the impact of fine-tuning, and ensures your LLM adheres to safety guidelines.
After having a career as a Data Scientist and Developer Advocate, Cheuk dedicated her work to the open-source community. Currently, she is working as a developer advocate for JetBrains. She has co-founded Humble Data, a beginner Python workshop that has been happening around the world. Cheuk also started and hosted a Python podcast, PyPodCats, which highlights the achievements of underrepresented members in the community. She has served the EuroPython Society board for two years and is now a fellow and director of the Python Software Foundation.