2026-09-11 –, Unconference
Search works great when you know what you're looking for. But what happens when a user types "sunset tours in Barcelona for a couple under 50 euros"? Traditional search pipelines break down.
We built an AI-powered discovery agent that lets users describe what they want in natural language and get relevant, ranked recommendations. In this talk, we'll walk through the production architecture: a multi-stage LangGraph pipeline that chains intent interpretation, parallel retrieval, LLM-based relevance filtering, and natural language synthesis.
We'll cover the practical trade-offs we made along the way, including a dual-LLM tier strategy (fast nano model for high-volume filtering, full model for nuanced interpretation) that keeps latency and cost in check without sacrificing quality.
We'll also share how we built evaluation infrastructure with 800+ deterministic tests and zero LLM-as-judge in CI, using a pass@k approach with pytest to handle output variability while keeping the test suite fast and reliable. You'll walk away with concrete patterns for shipping LLM-powered features to production: model tiering, deterministic evals, and graceful degradation by design.
This talk presents battle-tested patterns from building and shipping an LLM-powered discovery system to production. The focus is on three practical problems every team faces when moving LLM features from prototype to production: managing latency and cost at scale, building evaluation infrastructure you can trust, and designing for graceful failure.
Target audience: ML/AI Engineers and Data Scientists shipping LLM-powered features to production. Intermediate level, familiarity with LLMs and Python assumed.
Prior knowledge: Basic understanding of LLM APIs, Python, and pytest.
Key takeaways:
- How to implement a dual-LLM tier strategy to balance cost, latency, and quality
- A deterministic evaluation approach (pass@k with pytest) that avoids LLM-as-judge flakiness
- Production hardening patterns: scope gating, graceful degradation, every stage optional
Outline (30 min):
The problem: why natural language breaks traditional search (3 min)
- Live example: "sunset tours in Barcelona for a couple under 50 euros"
- Why keyword/semantic search can't handle filters, preferences, exclusions, and serendipity
- What's missing: reasoning, orchestration, conversational contextArchitecture overview (5 min)
- Multi-stage LangGraph pipeline: ContextResolver, Interpreter, Parallel Retrieval, NanoFilter, Grading, Synthesizer
- Intent interpretation: concrete example of parsing natural language into structured intent
- Routing logic: SIMPLE/DEEP/FILTERED paths based on query complexity
- Dual-LLM tier strategy (5 min)
- The problem: one model doesn't fit all stages
- Tier mapping: which model for which stage and why
- Results: 60% latency reduction, 4x cost savings, nano-tier recall 0.9 / precision 0.7
- Production hardening (3 min)
- Scope gating: reject non-travel queries early
- Graceful degradation: partial results are better than no results
- Every stage is optional: the pipeline adapts when components fail
- Evaluation infrastructure (10 min)
- Why not LLM-as-judge: non-deterministic evals in CI produce flaky tests, flaky tests kill trust
- pass@k approach: run each test case k=3 times via pytest-rerunfailures, pass if any succeeds
- 816 tests, 21 custom metrics, 4 datasets, 32 parallel workers, 0 LLM-as-judge calls in CI
- Real regression story: date inference change broke text extraction, caught before shipping
- Key lessons and Q&A (4 min)
- Right model per stage, deterministic evals over LLM-as-judge, graceful degradation by design, invest in eval infrastructure early
Giampaolo Casolla is a Senior Data Scientist at GetYourGuide, leveraging advanced machine learning and Generative AI to solve complex travel industry challenges. With expertise spanning areas like Safety, Risk, and Security, and strong skills in stats, Python, R, and cloud tech, he brings a diverse background to the role. Prior to GetYourGuide, Giampaolo developed award-winning ML solutions at Amazon and has a background in research with publications and conference presentations. At GetYourGuide, he's focused on integrating LLMs and GenAI into data products to drive innovation in travel technology.
Steven is a Software Engineer with 5+ years of experience building and productionizing machine learning systems that solve high-impact business problems. He has worked across domains including user acquisition, customer retention, content generation, supply optimization, and lead generation.
Currently at GetYourGuide, he is part of the Machine Learning and AI Platform team, where he focuses on enabling teams to independently build and operate production-grade LLM systems.
Outside of work, Steven co-organizes the AI in Production meetup in Berlin, a community dedicated to showcasing real-world AI applications across industries. In his free time, he enjoys exploring food and building tools he'd otherwise have to pay for.