Catching ML mistakes automatically with skore: from data scientists to agentic pipelines
Skore is an open source machine learning library that helps data scientists evaluate models and apply recommended practices throughout the ML development lifecycle. This talk dives into one of its core features: "checks," automated diagnostics that run on any scikit-learn-compatible estimator and flag issues with stable, actionable error codes. We'll see how checks catch common modeling pitfalls out of the box, how teams can write custom checks to encode their own business and compliance requirements, and how their structured output makes them a natural fit for agentic ML development loops.
Checks are accessible via the EstimatorReport object and come in two flavors:
Built-in checks automatically flag common modelling pitfalls, such as overfitting, underfitting, and class imbalance. Each is identified by a stable error code (e.g. SKD001, SKD002) that makes issues easy to track, mute, and communicate across a team.
Custom checks let you encode business requirements directly into the evaluation loop. For example, enforcing a minimum precision threshold on a fraud detection model, or ensuring a deployment-ready model meets internal fairness criteria. This makes skore a natural fit for teams that need to operationalize ML quality policies without reinventing the wheel.
We'll also look at how the structured, machine-readable output of checks makes them a powerful primitive in agentic ML pipelines. A coding agent can iterate on model development and use check results as feedback to guide its next step, catching mistakes that would otherwise slip through unnoticed.
Attendees will leave with a concrete understanding of how to use and extend skore checks, and a practical mental model for embedding automated diagnostics into both human and agent-driven ML workflows.
I'm a ML software engineer at probabl, doing core development for skore. I studied math and computer science and did a research masters in applied maths.