2026-09-10 –, Unconference
We needed to classify text into 5,000+ classes, each with detailed formal definitions, where correct labeling demands deep domain expertise. Manual annotation at this scale? Economically infeasible.
Our solution: use an LLM agent as an offline domain-expert labeler, then distill its knowledge into a classical ML model (gradient-boosted trees) that serves predictions in real time - approaching LLM-level quality at orders of magnitude less cost and latency.
We wrapped this in an automated active learning pipeline: production traffic surfaces novel inputs, the LLM labels them, the model retrains, and humans audit the results. Many cycles later, accuracy keeps climbing.
This talk presents the architecture, tradeoffs, and lessons from running this system in production. Data scientists and ML engineers who work with large label spaces or face labeling bottlenecks will walk away with a replicable pattern for using LLMs as labeling oracles - not as inference engines - to power fast, continuously improving ML systems.
Topic and Relevance
We built a production system that uses an LLM as an offline labeling engine for a 5,000+ class text classification problem, then distills that into a classical ML model (gradient-boosted trees) that meets real-time latency requirements. This is knowledge distillation - but with a classical model as the student instead of a smaller LLM.
This matters now because LLM quality and cost have reached a point where they can serve as reliable domain-expert labelers for complex, large-scale taxonomies - not just simple annotation tasks. This talk shares the architecture, tradeoffs, and lessons from running this system in production.
Outline (25 min + 5 min Q&A)
The Problem (5 min): 5,000+ class taxonomy where each class has formal definitions and correct labeling requires deep domain expertise. Manual labeling is economically infeasible. Unlabeled production data is abundant.
The LLM as Labeling Engine (7 min): Multi-step agentic workflows with external reference documents produce high-quality labels. We cover our quality assessment approach (qualitative review, subset testing, error analysis) and address the imperfect-labels objection: even fitting to the LLM's mistakes, matching its performance in a fast classical model is already a strong result.
Distilling into Production (6 min): Gradient-boosted trees trained on LLM labels achieve sub-real-time latency - this is what makes the product possible. We discuss the distillation framing and the path toward small fine-tuned transformers as the labeled data keep increasing.
Active Learning Loop (5 min): Automated pipeline: production traffic → frequency-prioritized novel input detection → LLM labeling → retraining. Models versioned with DVC/git, deployment through PRs with daily human review. Many of cycles, steady accuracy improvement.
When Does This Apply? (2 min): Open discussion on what makes a problem a fit for this pattern.
Audience Takeaways
The LLM-as-labeling-oracle pattern: when it applies, key design decisions, and what to watch out for - from a real production system.
How to think about imperfect LLM labels and manage labeling cost through frequency-based prioritization.
A practical approach to git-based active learning using (mostly) open-source tools (DVC, git, MLflow, LLM APIs).
Target Audience
Data scientists and ML engineers building products with large label spaces or labeling bottlenecks. Assumes basic supervised ML knowledge and general LLM familiarity.
Senior Data Scientist at Swap, based in the Netherlands. Background in computational science (MSc, Weizmann Institute of Science) with published research in machine learning for molecular dynamics, materials science, and atmospheric modeling. Previously worked on LLM-based systems, time series forecasting, and large-scale predictive pipelines. Enjoys making graphs and burning electrons.