Standard MLOps stacks assume you are operating tens or hundreds of models. What happens when you need to train, validate, and serve thousands of forecasting models on a weekly cadence, producing millions of forecasts for a national energy portfolio? Tools like MLflow start to buckle, per-model evaluation stops being meaningful, and the choice between one global model and many local ones becomes an architectural decision rather than a modeling one.
This talk is a systems architecture case study of the platform we built when off-the-shelf MLOps stopped working. We focus on three concrete, transferable patterns:
- Heterogeneous parallel training — running Spark ML global models and thousands of local models (via applyInPandas) in a single unified workflow, with a four-axis decision framework (data volume per entity, signal heterogeneity, cold-start behavior, operational cost) for routing each segment.
- A model registry that scales beyond MLflow — what we kept, what we dropped, and the schema that lets us manage versions, parameters, and artifacts for a vast portfolio without metadata-store collapse.
- Portfolio-level promotion gates — why per-model accuracy metrics mislead at scale, and how we run champion/challenger experiments with strict temporal integrity and aggregate decision criteria.
Attendees will leave with patterns they can apply to any large-scale time-series system, not just energy forecasting.