Pauline van Nies
Pauline is passionate about science and evaluating and optimizing language models for complex tasks. She is Tech Lead Data Science & AI at Sopra Steria and works for government organisations on efficient implementations of LLM and agentic usecases.
Session
Evaluating LLMs on standard benchmarks like EuroEval provides a baseline, but how do they handle tasks requiring genuine linguistic creativity like making clever associations and playing with words? This talk explores the challenge of solving cryptic crosswords - a complex task where last years state-of-the-art models only solve 25% of clues in a zero-shot setting.
We present a systematic approach to bridging this performance gap using different optimization techniques. We compare the reasoning capabilities of the latest models (gpt, claude, gemini, mistral and DeepSeek families) and demonstrate how to move beyond basic prompting. You will learn how to use DSPy for evaluation and automated prompt engineering plus if knowledge transfer or distillation - leveraging reasoning traces from "expert" models to guide smaller ones - can play a role in prompt optimization or finetuning.
And what if we provide the LLM with tools to look up e.g. synonyms and anagrams? We showcase a multi-agent architecture built with LangGraph that can complete entire Dutch cryptic crosswords (depending on the LLM powering it!).
Key takeaways include:
- A comparative analysis of reasoning vs. predict chat state-of-the-art models on creative linguistic tasks.
- A workflow for tracking experiments and optimizing prompts using MLflow and DSPy.
- Lessons on building agentic tool-use frameworks for complex language tasks.
- How these methods and strategies are transferable to business and even more complex puzzles.