BEGIN:VCALENDAR
VERSION:2.0
PRODID:-//pretalx//pretalx.com//pyconde-pydata-2026//talk//P8Y9TD
BEGIN:VTIMEZONE
TZID:CET
BEGIN:STANDARD
DTSTART:20001029T040000
RRULE:FREQ=YEARLY;BYDAY=-1SU;BYMONTH=10
TZNAME:CET
TZOFFSETFROM:+0200
TZOFFSETTO:+0100
END:STANDARD
BEGIN:DAYLIGHT
DTSTART:20000326T030000
RRULE:FREQ=YEARLY;BYDAY=-1SU;BYMONTH=3
TZNAME:CEST
TZOFFSETFROM:+0100
TZOFFSETTO:+0200
END:DAYLIGHT
END:VTIMEZONE
BEGIN:VEVENT
UID:pretalx-pyconde-pydata-2026-P8Y9TD@pretalx.com
DTSTART;TZID=CET:20260415T173500
DTEND;TZID=CET:20260415T180500
DESCRIPTION:You deploy an agent to automatically route incoming customer su
 pport tickets. At first\, it is a clear win: response times improve\, cust
 omers are happier\, and support teams finally get some rest.\n\nThen time 
 passes.\n\nNothing crashes. Dashboards stay green. No alerts fire. Yet the
  agent’s decisions slowly degrade first slightly\, then inconsistently\,
  and eventually becoming confidently wrong.\n\nThis is data drift.\n\nLLM-
 based agents in production operate in constantly changing environments. Pr
 oducts launch\, outages happen\, terminology evolves\, and priorities shif
 t. Unlike traditional ML models\, LLMs can produce plausible\, well-phrase
 d outputs even when they are incorrect\, making these failures difficult t
 o detect.\n\nIn this talk\, we focus on practical techniques for continuou
 sly evaluating and monitoring LLM-based agents after deployment. Using a s
 upport-ticket routing agent as an example\, we examine drift signals such 
 as increasing classification uncertainty\, spikes in fallback categories\,
  shifts in embedding distributions\, and growing disagreement with histori
 cal or human decisions.\n\nThe emphasis is not on training or prompt tunin
 g\, but on operating agents safely over time: detecting silent failures ea
 rly and knowing when intervention\, retraining\, or retirement is required
  before users notice.
DTSTAMP:20260523T180012Z
LOCATION:Ferrum [2nd Floor]
SUMMARY:The Day the Agent Started Lying (Politely) - Asya Melnik
URL:https://pretalx.com/pyconde-pydata-2026/talk/P8Y9TD/
END:VEVENT
END:VCALENDAR