When should an AI Agent say "I Don't Know"? Confidence, routing, and multi-turn evaluation in a LLM system
It's easy to get an LLM agent to call a tool and give a plausible answer. It's much harder to know when that answer is wrong, and whether a change actually made things better rather than just different.
This talk shares how we handled that in TAIA, TomTom's in-car conversational navigation backend: a stateful, tool-using Python/FastAPI service where an LLM interprets the driver's request and selects from deterministic TomTom APIs, never calculating a route itself.
TAIA returns a spoken response plus a structured command, and the in-car client renders the map and executes the action; session state loads from Azure Cosmos DB per request.
We skipped LangChain, the framework that hides prompting, tool calling, and memory behind its own abstractions, and built directly on Azure OpenAI clients, using OpenAI-compatible tool schemas over TomTom data.
Facts come from tool calls, not the model's own knowledge, and ambiguity is handled structurally: for "take me to King Street" with several nearby, TAIA returns options to the users instead of picking one and navigating.
We run a separate model per task: one for the main agentic loop, one that acknowledges the request while the main call runs in parallel, and different ones for each LLM-as-a-judge check; UX covers the remaining delay.
Evaluation is layered rather than one metric: a pytest-bdd/Gherkin suite covering different scenarios and mixing deterministic checks with zero-temperature LLM judges, with every judge call's tokens, cost, and latency exported to Azure Data Explorer.
The patterns aren't car-specific — they apply to any agent using real tools where mistakes matter — and the same TomTom APIs are available through TomTom's MCP server and Agent Toolkit.
Target Audience:
It's aimed at people who now have to ship agents rather than prototype them, and it covers the part they struggle with most— evaluation and reliability. The same TomTom APIs are public through TomTom's MCP server and Agent Toolkit, so anyone can try this afterwards.
Required Background
Familiarity with LLMs, basic agent workflows, and production ML or software systems will be helpful.
Audience Takeaways
People leave with patterns they can reuse: return options instead of guessing when a request is ambiguous, get facts from tools rather than the model, pick a different model per task to control latency and cost, and build layered evaluations that show whether a change actually helped. All of it works in plain Python.
Outline:
- Intro — why plausible answers are not enough — 4min
- TAIA architecture — tool-using Python/FastAPI backend, TomTom APIs, session state, and client handoff — 8min
- Grounding and ambiguity — when the agent should act, ask, or say it does not know — 7min
- Model routing and latency — separate models, instant acknowledgements, cost, and UX delay-masking — 5min
- Evaluation — pytest-bdd/Gherkin scenarios, deterministic checks, LLM judges, and telemetry — 7min
- Takeaways and Q&A — what generalizes beyond cars, TomTom MCP/Agent Toolkit, and audience questions — 7min
Leading a team delivering data and analytics solutions that enable TomTom’s Product and Engineering organizations to operate in a more data‑driven way.
Overseeing the development of TomTom’s agentic products, including TomTom AI Agent, TomTom MCP, Agent Toolkit as well as additional emerging agentic capabilities entering the market.
With a background in data science and ML engineering, I focus on combining advanced AI with scalable engineering to build intelligent, future‑ready products.