BEGIN:VCALENDAR
VERSION:2.0
PRODID:-//pretalx//pretalx.com//pydata-amsterdam2026//speaker//VU7CFV
BEGIN:VTIMEZONE
TZID:Europe/Amsterdam
BEGIN:DAYLIGHT
DTSTART:20250911T000000
TZNAME:CEST
TZOFFSETFROM:+0200
TZOFFSETTO:+0200
END:DAYLIGHT
BEGIN:STANDARD
DTSTART:20251026T030000
RDATE:20261025T030000
TZNAME:CET
TZOFFSETFROM:+0200
TZOFFSETTO:+0100
END:STANDARD
BEGIN:DAYLIGHT
DTSTART:20260329T030000
RDATE:20270328T030000
TZNAME:CEST
TZOFFSETFROM:+0100
TZOFFSETTO:+0200
END:DAYLIGHT
END:VTIMEZONE
BEGIN:VEVENT
SUMMARY:When should an AI Agent say "I Don't Know"? Confidence\, routing\,
  and multi-turn evaluation in a LLM system - Tara Farzami
DTSTART;TZID=Europe/Amsterdam:20260911T100500
DTEND;TZID=Europe/Amsterdam:20260911T105000
DTSTAMP:20260911T153628Z
UID:pretalx-pydata-amsterdam2026-ZXZCFJ@pretalx.com
DESCRIPTION:It's easy to get an LLM agent to call a tool and give a plausi
 ble answer. It's much harder to know when that answer is wrong\, and wheth
 er a change actually made things better rather than just different. \nThis
  talk shares how we handled that in TAIA\, TomTom's in-car conversational 
 navigation backend: a stateful\, tool-using Python/FastAPI service where a
 n LLM interprets the driver's request and selects from deterministic TomTo
 m APIs\, never calculating a route itself. \nTAIA returns a spoken respons
 e plus a structured command\, and the in-car client renders the map and ex
 ecutes the action\; session state loads from Azure Cosmos DB per request. 
 \nWe skipped LangChain\, the framework that hides prompting\, tool calling
 \, and memory behind its own abstractions\, and built directly on Azure Op
 enAI clients\, using OpenAI-compatible tool schemas over TomTom data. \nFa
 cts come from tool calls\, not the model's own knowledge\, and ambiguity i
 s handled structurally: for "take me to King Street" with several nearby\,
  TAIA returns options to the users instead of picking one and navigating. 
 \nWe run a separate model per task: one for the main agentic loop\, one th
 at acknowledges the request while the main call runs in parallel\, and dif
 ferent ones for each LLM-as-a-judge check\; UX covers the remaining delay.
  \nEvaluation is layered rather than one metric: a pytest-bdd/Gherkin suit
 e covering different scenarios and mixing deterministic checks with zero-t
 emperature LLM judges\, with every judge call's tokens\, cost\, and latenc
 y exported to Azure Data Explorer. \nThe patterns aren't car-specific — 
 they apply to any agent using real tools where mistakes matter — and the
  same TomTom APIs are available through TomTom's MCP server and Agent Tool
 kit.
LOCATION:Entropy
URL:https://pretalx.com/pydata-amsterdam2026/talk/ZXZCFJ/
END:VEVENT
END:VCALENDAR