2026-09-11 –, Room 2 (350)
Serving user-specific ML models at scale while satisfying low-latency requirements is non-trivial, especially when the model training code is regularly updated, making models trained weeks apart require potentially incompatible environments. How do you build a scalable inference pipeline with per-user models while still moving fast, when the runtime keeps evolving? In this talk, I’ll show how our small team built and maintains a Python inference pipeline that can serve thousands of user-specific models from a single service using a dynamic model loading approach. There is still one challenge to tackle: enabling training code and dependency update without risking incompatible inference environments. We’ll compare different strategies and explain the solution we chose to stay agile while preserving a smooth UX and DevEx. Finally, we will walk through the end-to-end architecture built using open source tools such as MLflow, MLServer, KServe, and supporting both local development and production-scale deployments.
Most ML systems serve a handful of general models to all users. On the other hand, a growing class of products instead trains a model per user on their own data, which is what the Atinary SDLabs platform does: these per-user models power our Bayesian Optimization algorithms that recommend the next experiments to run. With training code updated roughly every two weeks and an inference latency budget of a few seconds, how can we persist and serve these models in scalable ways while satisfying product requirements?
To build such a system, one could think of deploying a dedicated inference service per user. However this would quickly become operationally infeasible (at least for us). What about a shared inference environment then? Well, at the same time, the training code might be regularly improved, meaning that models trained just weeks apart may depend on incompatible runtimes. This makes shared inference environments risky or even impossible...
In this talk, I'll walk you through a real production system built under these constraints. We will first go through different architectural options and how we evaluated strategies such as enforcing strict backward compatibility, retraining only on breaking changes, or building large monolithic inference images that support every historical model version. For each option, I'll discuss not only technical feasibility, but also operational cost and impact on developer experience.
Finally, I'll explain the solution we ultimately chose for our scalability and latency requirements: a single inference service that dynamically loads models at request time, paired with an overnight post-release retraining job that proactively refreshes models for our most active users. I will then go through the open source tech stack: MLflow for model persistence, MLServer as the serving engine, the Open Inference Protocol as standardized interface, and KServe for Kubernetes-native scaling. The same setup supports both local development and production deployments, enabling fast iteration without compromising reliability.
This talk is aimed at ML engineers, platform engineers, and applied data scientists who are building or operating Python-based ML systems in production and want practical insights on handling evolving runtimes, multi-model serving, and real-world constraints.
Currently a Machine Learning Engineer at Atinary Technologies in Lausanne, Switzerland. Redesigned the training pipeline and introduced an inference infrastructure for the SDLabs platform. Now part of the research team to improve our bayesian optimization and transfer learning algorithms.
Before, I worked as an ML Research Intern at Bose and Logitech on lightweight, causal speech enhancement and source separation under strict latency and memory constraints.
Before that I obtained a MSc in Computer Science from EPFL, with focus on Machine Learning and Digital Signal Processing.
I travel and produce electronic music during my free time!
rayan.daodnathoo.com