Answers you can question: building a verifiable AI analytics agent
At Manychat, we built an analytics agent on top of Claude Code that lets non-data specialist teammates ask data questions in plain English. You type /ask followed by a question; the agent clarifies it if needed, routes it to the right business domain, queries curated data marts, and returns the numbers plus everything an analyst needs to verify them.
Connecting an agent to a data warehouse is easy, but without the right context, it can provide answers that seem plausible rather than correct. And a confidently wrong number doesn’t stay in the terminal - it ends up in someone’s all-hands deck.
This talk covers the layer that closes that gap: how we decide what the agent sees, how we keep that context from going stale, how the agent determines how much confidence to place in its own answer, and how we detect when a change quietly breaks something.
Description
An architecture talk about an AI analytics agent that non-technical people at Manychat use daily, and the design decisions that make its answers checkable instead of merely plausible.
It covers why raw warehouse access is the wrong default for an LLM agent and what must sit between the question and the data: curated context that stays fresh and is loaded selectively for each question, an explicit way to determine how much an answer can be trusted, and a way to catch regressions when any of it changes. It’s an experience report as much as an architecture talk: what we learned while building it, what we still want to improve, and why we’ve kept it in-house so far.
Solution limitations
- Confidently wrong answers are cheaper to catch, not eliminated. The design lowers the cost of verification but does not guarantee correctness.
- Context freshness is enforced only where we built tooling for it, the semantic layer and the BI mirror. Everywhere else it is still manual.
- Answer validation is still human. Anything that matters gets checked by an analyst.
Audience takeaways
- The confidence that AI analytics is something a small team can build in-house today and start experimenting with
- What an answer has to carry to make it verifyable: the numbers, the views it used, the grain, etc, and the SQL to re-run.
- How to keep context fresh without relying on anyone remembering to update it: metadata files generated by tooling rather than written by hand, a CI check that blocks a pull request when the SQL changed and the documentation did not.
- Why less context produces better answers, and the mechanism we use: per-domain metadata that is pulled in only when the question matches it, instead of one large context loaded for every question.
- Ways to evaluate a system whose output is never identical twice and detect when changes introduce regressions.
The talk is for data engineers, analytics engineers, and analysts who are asked to put an agent in front of a warehouse. Prior knowledge: a rough understanding of how analytics work is organized.
Outline
- Introduction (2 min)
- Why raw warehouse access fails (3 min)
- The architecture + demo (4 min)
- Context layers: views metadata, always-on rules, per-question routing, and trust levels (7 min)
- Evaluation and observability: the golden set, analyst notifications, and traces (3 min)
- Build vs. buy (3 min)
- Learnings, limitations, and open problems (3 min)
Picked up a Master's in Bioengineering and Bioinformatics, followed by 15+ years as a backend/database engineer in Healthcare and Science domain. Currently a Data Engineer at Manychat, bridging platform data engineering with ML/AI.