BEGIN:VCALENDAR
VERSION:2.0
PRODID:-//pretalx//pretalx.com//pydata-amsterdam2026//speaker//KECWAL
BEGIN:VTIMEZONE
TZID:CET
BEGIN:STANDARD
DTSTART:20001029T040000
RRULE:FREQ=YEARLY;BYDAY=-1SU;BYMONTH=10
TZNAME:CET
TZOFFSETFROM:+0200
TZOFFSETTO:+0100
END:STANDARD
BEGIN:DAYLIGHT
DTSTART:20000326T030000
RRULE:FREQ=YEARLY;BYDAY=-1SU;BYMONTH=3
TZNAME:CEST
TZOFFSETFROM:+0100
TZOFFSETTO:+0200
END:DAYLIGHT
END:VTIMEZONE
BEGIN:VEVENT
UID:pretalx-pydata-amsterdam2026-XUC7LQ@pretalx.com
DTSTART;TZID=CET:20260911T145500
DTEND;TZID=CET:20260911T152500
DESCRIPTION:LLM agents are reaching production faster than teams can evalua
 te them. A data-analysis agent that runs the right query but reports the w
 rong number\, or returns the right number via a trajectory full of fabrica
 ted tool calls\, passes superficial testing and fails in production.\n\nTh
 is talk walks through evaluating such an agent end-to-end. Our running exa
 mple: a data-analysis agent answering questions over a business dataset. W
 e show how to grade three dimensions that agent evaluation requires and si
 ngle-shot LLM evaluation ignores: final response\, trajectory\, and state 
 changes.\n\nWe cover the full lifecycle:\n\n1. Bootstrapping evaluation fr
 om 50 hand-reviewed examples when you have no labels.\n2. Aligning an LLM-
 as-a-judge to human judgment with the same rigor you'd apply to outsourced
  annotators: dev/test splits\, inter-rater agreement\, Cohen's kappa.\n3. 
 Scaling to continuous online evaluation with CI integration\, error analys
 is\, and prompt optimization driven by natural-language feedback.\n\nWe al
 so cover what we got wrong in earlier iterations and what we'd do differen
 tly today.\n\nAttendees will leave with a process they can run on their ow
 n agent next week\, and a clear rule for when to trust an automated judge 
 at scale\, and when to stop.
DTSTAMP:20260710T145538Z
LOCATION:Room 2 (350)
SUMMARY:Evaluating Agents at Scale: From 50 Examples to a Production Flywhe
 el - Bauke  Brenninkmeijer
URL:https://pretalx.com/pydata-amsterdam2026/talk/XUC7LQ/
END:VEVENT
END:VCALENDAR