PyData Amsterdam 2026

Coding agents have improved over the past year to the point where they can iterate on a codebase with minimal supervision. ML projects look like a natural fit: the iteration loop is well defined and the code surface is small compared to a full application. ML is not a single-trajectory task, though. Picking the right configuration involves exploration: random sampling over hyperparameters, architectural variants, data subsets. Domain knowledge alone rarely settles the question, and the exploration is repetitive, which is why people want to automate it. A recent example of this direction is Karpathy's autoresearch, which showed that a coding agent given an .md prompt can spin up its own worktrees and track experiments in a CSV. However, the trajectory it explored was single and shallow, and most of the modifications it tried could have been handled by a grid search. Real autonomous experimentation needs more structure than that.

The natural next step is to add a second LLM agent on top to coordinate the others, but in practice this does not hold up. The orchestrating agent drifts in the same way the workers do, just on a longer horizon: after a few dozen experiments it forgets which directions were already tried, mixes unrelated changes into a single step, or commits to a branch that the metrics do not justify. The coordination problem cannot be solved by adding more prompting.

The approach we'll present takes the opposite direction. Coordination is handled by deterministic, non-LLM code, and the LLM is used only where open-ended generation is actually needed. The orchestrator runs experimentation as a tree search in which each node represents a hypothesis-constrained modification to a reproducible baseline, and modifications are restricted to a single aspect per step so that the resulting metric change can be attributed.

From there, we walk through what this looks like in practice. We start with how the LLM work inside the tree is split by role: a proposing node generates candidate modifications, an implementing node turns a chosen proposal into code, and a debugging node is invoked when an experiment fails to run. Each role operates with a fresh context and a narrow scope rather than a single agent carrying the full history. We then cover the supporting infrastructure that makes the tree practical in real projects: git worktrees for branch isolation and parallel execution, and an anti-repetition memory that prevents the same hypothesis from coming back under different wording.

Additionally, we cover two components that are extremely useful for these kind of systems. The first is the path for prior results to flow back into fresh agent contexts. For this we use a real experiment tracking backend exposed through a token-efficient query interface, which doubles as the surface where a human can step in to inspect the results. The second is the choice not to reimplement the coding CLI itself: Claude Code, Codex, and similar tools are dispatched as replaceable backends, and can be easily swapped or even combined with almost no code changes.

We close with lessons from running this on real experimentation work, including where the deterministic structure helped, where it got in the way, and where a well-specified hyperparameter sweep is still the better tool.

Outline:

• Why ML experimentation breaks coding agents (5 min)
  • Differences between Software Development and ML
  • autoresearch and its limits
• What goes wrong when an LLM orchestrates other LLMs (4 min)
• The deterministic orchestrator (8 min)
  • Tree search over hypothesis-constrained modifications
  • Role-separated nodes: proposing, implementing, debugging
  • Importance of fresh context and narrow scope per node
• Supporting infrastructure (5 min)
  • Git worktrees for branch isolation and parallel execution
  • Anti-repetition memory
  • Tocken-efficient agent interface for experiment tracking tools
• Adoption and trade-offs (3 min)
  • Human-in-the-loop
  • Harness-agnostic integration with Claude Code, Codex, and similar
  • Lessons from real work, and when a regular Optuna search is still better
• Q&A (5 min)

Target audience:

The talk is aimed at ML engineers, applied researchers, and data scientists who have tried using coding agents to drive experimentation, or are considering it, and at platform engineers who are asked to support such workflows. Familiarity with a standard training and evaluation loop is assumed. No prior experience with multi-agent systems is required.

Learning outcomes:

By the end of the talk, attendees will have a clear picture of how to build a system for autonomous ML experimentation on top of widely used coding harnesses like Claude Code, Codex (or any other), as well as an understanding of the limitations of such a system.