BEGIN:VCALENDAR
VERSION:2.0
PRODID:-//pretalx//pretalx.com//pydata-amsterdam2026//speaker//8FZRES
BEGIN:VTIMEZONE
TZID:CET
BEGIN:STANDARD
DTSTART:20001029T040000
RRULE:FREQ=YEARLY;BYDAY=-1SU;BYMONTH=10
TZNAME:CET
TZOFFSETFROM:+0200
TZOFFSETTO:+0100
END:STANDARD
BEGIN:DAYLIGHT
DTSTART:20000326T030000
RRULE:FREQ=YEARLY;BYDAY=-1SU;BYMONTH=3
TZNAME:CEST
TZOFFSETFROM:+0100
TZOFFSETTO:+0200
END:DAYLIGHT
END:VTIMEZONE
BEGIN:VEVENT
UID:pretalx-pydata-amsterdam2026-YPZA7R@pretalx.com
DTSTART;TZID=CET:20260910T150500
DTEND;TZID=CET:20260910T153500
DESCRIPTION:We needed to classify text into 5\,000+ classes\, each with det
 ailed formal definitions\, where correct labeling demands deep domain expe
 rtise. Manual annotation at this scale? Economically infeasible.\nOur solu
 tion: use an LLM agent as an offline domain-expert labeler\, then distill 
 its knowledge into a classical ML model (gradient-boosted trees) that serv
 es predictions in real time - approaching LLM-level quality at orders of m
 agnitude less cost and latency.\nWe wrapped this in an automated active le
 arning pipeline: production traffic surfaces novel inputs\, the LLM labels
  them\, the model retrains\, and humans audit the results. Many cycles lat
 er\, accuracy keeps climbing.\nThis talk presents the architecture\, trade
 offs\, and lessons from running this system in production. Data scientists
  and ML engineers who work with large label spaces or face labeling bottle
 necks will walk away with a replicable pattern for using LLMs as labeling 
 oracles - not as inference engines - to power fast\, continuously improvin
 g ML systems.
DTSTAMP:20260710T150459Z
LOCATION:Unconference
SUMMARY:Distilling LLMs into Classical ML for 5\,000+ Classes - Oz Mendelso
 hn
URL:https://pretalx.com/pydata-amsterdam2026/talk/YPZA7R/
END:VEVENT
END:VCALENDAR