2026-07-16 –, Johnson Great Room
How do we build competent data analysis agents? Data analysis requires a willingness to pause, question conclusions, and dig into subtleties. Frontier LLMs, however, are optimized to push tasks toward completion, not to slow down when something seems off. This tendency works well for coding agents, where success is often verifiable. But for data analysis, verification is more complicated, and autonomous work by the agent can be at odds with the spirit of the discipline. Drawing on our experience building data analysis agents, we'll share evaluations that expose where LLM-driven analysis goes wrong and design patterns that keep analyses correct, transparent, and reproducible.
LLM-powered agents are increasingly used for software development and data analysis. However, LLMs are non-deterministic, have uneven competencies, and can lack important context for realistic tasks. For software development, models can typically leverage tight feedback loops. It is often clear if code accomplishes its goal, and the model can also write both code and tests for that code, using the test results to iterate on its work. For data analysis, however, it’s often less clear if the model has done the task well or provided a correct result.
How, then, do we make competent data analysis agents? In this talk, we will discuss strategies for creating data analysis agents that produce correct, transparent, and reproducible results. We will use examples from Posit Assistant, Posit’s general-purpose coding and data analysis agent. The intended audience includes scientists or data practitioners interested in using AI in data analysis workflows.
First, we will discuss the importance of empirical evaluation. Because LLM capabilities can be difficult to predict, we created a series of evaluations, some using the Python library Inspect, to measure the capabilities of the skills we care about. These evaluations help us make decisions about model choice, tool design, and prompting, as well as identify any critical issues in the models’ abilities to carry out data science tasks. As an example, we will discuss bluffbench, an evaluation that measures LLMs’ ability to interpret plots that conflict with their priors. We will also discuss a developmental benchmark that measures agents’ ability to surface subtle data quality issues across long contexts.
Second, we will discuss design choices to make agent-assisted analyses transparent and reproducible. Data analysis involves a variety of tasks, and different tasks require different levels of human awareness, input, and understanding. For example, exploratory data analysis still typically requires input and understanding from the user by nature of the task. Thus, when doing EDA, our agents produce briefer responses and ask the user for more input. For coding tasks with clear goals, however, we can often trust the agents to act more autonomously.
Data analysis agents introduce both risks and opportunities for rigorous data analysis. Our aim for this talk is to introduce practical guidance for evaluating and creating data analysis agents that can be integrated into scientific workflows, while preserving accuracy, transparency, and reproducibility.
Related work:
- Bluffbench and plot interpretation: Bluffbench repo, Introducing bluffbench, and How well do LLMs interpret plots?
- Introducing Databot and Databot is not a flotation device. Posit Assistant will be released in March and so does not yet have public documentation.
- Next edit suggestions (code completion) evaluations
- Evidence of public speaking ability:
Sara Altman is a developer advocate on the AI Core team at Posit, where she focuses on how AI can be effectively and thoughtfully used for data science. She co-authors the Posit AI newsletter with Simon Couch. Previously, she helped build Posit Academy and taught data science and R at Stanford.
Simon Couch builds tools that make the work of data science more joyful and effective. As an engineer on the AI Core Team at Posit, his work spans coding assistants, model evaluations, and next-edit-suggestion systems. Drawing on his background in statistics, Simon spent several years authoring and maintaining core packages in the open-source tidymodels framework—like stacks, broom, and infer—before shifting his focus to LLMs. He blogs about his work at simonpcouch.com.