2026-09-11 –, Room 1 (170)
Standard MLOps stacks assume you are operating tens or hundreds of models. What happens when you need to train, validate, and serve thousands of forecasting models on a weekly cadence, producing millions of forecasts for a national energy portfolio? Tools like MLflow start to buckle, per-model evaluation stops being meaningful, and the choice between one global model and many local ones becomes an architectural decision rather than a modeling one.
This talk is a systems architecture case study of the platform we built when off-the-shelf MLOps stopped working. We focus on three concrete, transferable patterns:
- Heterogeneous parallel training — running Spark ML global models and thousands of local models (via applyInPandas) in a single unified workflow, with a four-axis decision framework (data volume per entity, signal heterogeneity, cold-start behavior, operational cost) for routing each segment.
- A model registry that scales beyond MLflow — what we kept, what we dropped, and the schema that lets us manage versions, parameters, and artifacts for a vast portfolio without metadata-store collapse.
- Portfolio-level promotion gates — why per-model accuracy metrics mislead at scale, and how we run champion/challenger experiments with strict temporal integrity and aggregate decision criteria.
Attendees will leave with patterns they can apply to any large-scale time-series system, not just energy forecasting.
This is a systems architecture talk, not a forecasting tutorial or a general MLOps overview. We will not cover modeling techniques, domain-specific feature engineering, or monitoring dashboards in depth — the focus is on the architectural decisions that make portfolio-scale forecasting operationally viable.
Introduction & Problem Framing (4 minutes)
- The scale: ~100 thousand models, 20 million forecasts per week, 30 TB of time-series data, weekly retraining SLA.
- Why this is a systems problem, not a modeling problem: the bottleneck is orchestration, registry, and evaluation — not algorithm choice.
- Where standard stacks break: MLflow metadata limits, per-model CI/CD, individual-model evaluation.
- Talk roadmap: three patterns, one architecture diagram, no live demo.
Architectural Overview (3 minutes)
- A single diagram of the end-to-end platform.
- Brief mention of supporting infrastructure (data lake, feature store, inference, monitoring) — flagged as out of scope for depth, available in Q&A.
Pattern 1: Heterogeneous Parallel Training (7 minutes)
- The global-vs-local question at portfolio scale: when does one Spark ML model beat thousands of local models, and vice versa?
- Our decision framework — four axes that determine the global-vs-local split:
- Data volume per entity — sparse history → global; deep per-series history → local.
- Signal heterogeneity — homogeneous behavior across entities → global; strongly divergent patterns (EV charging, heat pumps, prosumers) → local.
- Cold-start behavior — new or weakly observed entities must forecast from day one → global; warm historical context available → local.
- Operational cost — training/registry/serving budget is tight → global; added complexity justified by better local decisions → local.
- Unified workflow architecture: Spark ML for global models and applyInPandas for local models, orchestrated in a single PySpark job.
- What generalizes: the routing logic, the partitioning strategy, the failure-isolation pattern.
Pattern 2: A Registry That Scales Beyond MLflow (6 minutes)
- Where managed MLflow broke for us at scale:
- Retrieval latency — API calls to fetch models degrade to tens of seconds to several minutes once the registry holds 50k+ model versions, making batch inference pipelines unacceptably slow.
- Artifact storage creep — even at ~1 KB per serialized model, 100k models registered weekly compounds to ~5 TB/year of artifacts with no built-in lifecycle control; storage costs scale superlinearly due to metadata overhead.
- Platform limits — Databricks workspace resource limits become a hard constraint at this scale.
- What we kept from the MLflow mental model (experiment tracking, versioning concepts) and what we replaced (artifact storage, metadata queries, model retrieval).
- Our registry schema: versioning, parameter tracking, artifact storage, and lineage at portfolio scale.
- What generalizes: the schema design, the read/write access patterns, the integration points with training and inference.
Pattern 3: Portfolio-Level Promotion Gates (7 minutes)
- Why per-model metrics can be misleading: a challenger model may outperform at the portfolio level while still underperforming on individual local level.
- Distributed backtesting with strict temporal integrity: parallel folds across thousands of entities, no leakage.
- Champion/challenger design: experimental pipelines running in parallel with production on identical folds.
- Promotion criteria: portfolio-aggregated metrics + operational robustness + an explainable driver for the improvement — not just better numbers.
- What generalizes: the aggregation logic, the promotion checklist, the parallel-pipeline architecture.
Takeaways & Q&A (3 minutes)
- The three patterns, restated as a checklist.
- The shift from a model-centric to a system-centric view of ML.
- Pointers to topics deferred to Q&A: monitoring, resilience, retraining strategy, feature store.
Machine Learning Engineer with a strong focus on building scalable and robust ML/AI platforms and systems.