Oz Mendelsohn
Senior Data Scientist at Swap, based in the Netherlands. Background in computational science (MSc, Weizmann Institute of Science) with published research in machine learning for molecular dynamics, materials science, and atmospheric modeling. Previously worked on LLM-based systems, time series forecasting, and large-scale predictive pipelines. Enjoys making graphs and burning electrons.
Session
We needed to classify text into 5,000+ classes, each with detailed formal definitions, where correct labeling demands deep domain expertise. Manual annotation at this scale? Economically infeasible.
Our solution: use an LLM agent as an offline domain-expert labeler, then distill its knowledge into a classical ML model (gradient-boosted trees) that serves predictions in real time - approaching LLM-level quality at orders of magnitude less cost and latency.
We wrapped this in an automated active learning pipeline: production traffic surfaces novel inputs, the LLM labels them, the model retrains, and humans audit the results. Many cycles later, accuracy keeps climbing.
This talk presents the architecture, tradeoffs, and lessons from running this system in production. Data scientists and ML engineers who work with large label spaces or face labeling bottlenecks will walk away with a replicable pattern for using LLMs as labeling oracles - not as inference engines - to power fast, continuously improving ML systems.