2026-09-10 –, Room 2 (350)
Evaluating LLMs on standard benchmarks like EuroEval provides a baseline, but how do they handle tasks requiring genuine linguistic creativity like making clever associations and playing with words? This talk explores the challenge of solving cryptic crosswords - a complex task where last years state-of-the-art models only solve 25% of clues in a zero-shot setting.
We present a systematic approach to bridging this performance gap using different optimization techniques. We compare the reasoning capabilities of the latest models (gpt, claude, gemini, mistral and DeepSeek families) and demonstrate how to move beyond basic prompting. You will learn how to use DSPy for evaluation and automated prompt engineering plus if knowledge transfer or distillation - leveraging reasoning traces from "expert" models to guide smaller ones - can play a role in prompt optimization or finetuning.
And what if we provide the LLM with tools to look up e.g. synonyms and anagrams? We showcase a multi-agent architecture built with LangGraph that can complete entire Dutch cryptic crosswords (depending on the LLM powering it!).
Key takeaways include:
- A comparative analysis of reasoning vs. predict chat state-of-the-art models on creative linguistic tasks.
- A workflow for tracking experiments and optimizing prompts using MLflow and DSPy.
- Lessons on building agentic tool-use frameworks for complex language tasks.
- How these methods and strategies are transferable to business and even more complex puzzles.
Outline
- 0-5 min: Introduction Language models and Cryptic Crosswords, benchmarking different LLMs
- 5 - 10 min: Is it possible to improve the LLM performance by expert guided few-shot learning using prompt optimization?
- 10 -15 min: Is it possible to improve the LLM performance by knowledge distillation using finetuning?
- 15 - 20 min: LLMs + tools: Agent framework choice and setup
- 20 -25 min: Showcase Cryptic Crosswords solving by agents & Comparison LLMs
- 25 - 30 min: Lessons learned and explanation how transferable is the framework to business related tasks and even more complex puzzles (AIVD Christmas puzzle?)
We are interested in evaluating and optimizing language models on a complex task that requires linguistic insight and creativity. Cryptic crosswords contain clues with misleading surface reading, whose solutions require disambiguation of wordplay. Hence, a language model needs to conduct multi-step (iterative) reasoning by extracting and/or identifying the relevant components of a cryptic clue that lead to the final answer. Our aim is to get a better understanding of the capabilities and bottlenecks of language models, as well as an interesting insight in how language models solve a task that in humans is described as a feeling of ‘the solution just falls in place’.
Thus, we set up an evaluation pipeline for language models to test their ability to solve cryptic clues (Dutch and English datasets). In this presentation, we will share the results of comparing different LLMs and optimization approaches on performing this specific task. We apply standard optimization search methods like adjusting the temperature and using Chain-of-thought, adding domain expertise and the few-shot approach. Additionally, we apply automated prompt engineering using the DSPy framework. We will showcase this approach of systematically experimenting, tracking results (mlflow) and optimizing a language model and prompt.
Next, we will show if an LLM can learn to solve a cryptic clue, by either knowledge distillation from expert reasoning traces as a guided few-shot approach for in-context learning or by transferring the behaviour of an LLM teacher to an (initially worse performing) LLM student with finetuning.
Finally, we simulate how LLM agents can solve a cryptic crossword together, using different strategies that humans would do naturally. In the agent framework, tools such as look up of anagrams and synonyms are provided to the react agent to help solving the clues. The implementation of the puzzle agent framework and the choice for its technical framework (LangGraph over Google ADK) is elaborated on and the challenges to overcome. We will show with a demo how the puzzle agent framework (depending on the LLM it is powered by) in real-time solves a cryptic crossword.
We summarize the conclusions and comment shortly on how these insights relate to business projects. We also give an outlook on the question: how transferable is this puzzle agent framework to even more complex puzzles? In the Netherlands the holy grail of puzzle solving are the AIVD Christmas puzzles which contains a.o. language puzzles. We will comment on the possibility to solve those with the puzzle agent framework and future approaches.
Pauline is passionate about science and evaluating and optimizing language models for complex tasks. She is Tech Lead Data Science & AI at Sopra Steria and works for government organisations on efficient implementations of LLM and agentic usecases.