Systems for Scale: Architecting a Nationwide Energy Forecasting Platform
Standard MLOps stacks assume you are operating tens or hundreds of models. What happens when you need to train, validate, and serve thousands of forecasting models on a weekly cadence, producing millions of forecasts for a national energy portfolio? Tools like MLflow start to buckle, per-model evaluation stops being meaningful, and the choice between one global model and many local ones becomes an architectural decision rather than a modeling one.
This talk is a systems architecture case study of the platform we built when off-the-shelf MLOps stopped working. We focus on three concrete, transferable patterns:
- Heterogeneous parallel training — running Spark ML global models and thousands of local models (via applyInPandas) in a single unified workflow, with a four-axis decision framework (data volume per entity, signal heterogeneity, cold-start behavior, operational cost) for routing each segment.
- A model registry that scales beyond MLflow — what we kept, what we dropped, and the schema that lets us manage versions, parameters, and artifacts for a vast portfolio without metadata-store collapse.
- Portfolio-level promotion gates — why per-model accuracy metrics mislead at scale, and how we run champion/challenger experiments with strict temporal integrity and aggregate decision criteria.
Attendees will leave with patterns they can apply to any large-scale time-series system, not just energy forecasting.
This is a systems architecture talk, not a forecasting tutorial or a general MLOps overview. We will not cover modeling techniques, domain-specific feature engineering, or monitoring dashboards in depth — the focus is on the architectural decisions that make portfolio-scale forecasting operationally viable.
Introduction & Problem Framing (4 minutes)
- The scale: ~100 thousand models, 20 million forecasts per week, 30 TB of time-series data, weekly retraining SLA.
- Why this is a systems problem, not a modeling problem: the bottleneck is orchestration, registry, and evaluation — not algorithm choice.
- Where standard stacks break: MLflow metadata limits, per-model CI/CD, individual-model evaluation.
- Talk roadmap: three patterns, one architecture diagram, no live demo.
Architectural Overview (3 minutes)
- A single diagram of the end-to-end platform.
- Brief mention of supporting infrastructure (data lake, feature store, inference, monitoring) — flagged as out of scope for depth, available in Q&A.
Pattern 1: Heterogeneous Parallel Training (7 minutes)
- The global-vs-local question at portfolio scale: when does one Spark ML model beat thousands of local models, and vice versa?
- Our decision framework — four axes that determine the global-vs-local split:
- Data volume per entity — sparse history → global; deep per-series history → local.
- Signal heterogeneity — homogeneous behavior across entities → global; strongly divergent patterns (EV charging, heat pumps, prosumers) → local.
- Cold-start behavior — new or weakly observed entities must forecast from day one → global; warm historical context available → local.
- Operational cost — training/registry/serving budget is tight → global; added complexity justified by better local decisions → local.
- Unified workflow architecture: Spark ML for global models and applyInPandas for local models, orchestrated in a single PySpark job.
- What generalizes: the routing logic, the partitioning strategy, the failure-isolation pattern.
Pattern 2: A Registry That Scales Beyond MLflow (6 minutes)
- Where managed MLflow broke for us at scale:
- Retrieval latency — API calls to fetch models degrade to tens of seconds to several minutes once the registry holds 50k+ model versions, making batch inference pipelines unacceptably slow.
- Artifact storage creep — even at ~1 KB per serialized model, 100k models registered weekly compounds to ~5 TB/year of artifacts with no built-in lifecycle control; storage costs scale superlinearly due to metadata overhead.
- Platform limits — Databricks workspace resource limits become a hard constraint at this scale.
- What we kept from the MLflow mental model (experiment tracking, versioning concepts) and what we replaced (artifact storage, metadata queries, model retrieval).
- Our registry schema: versioning, parameter tracking, artifact storage, and lineage at portfolio scale.
- What generalizes: the schema design, the read/write access patterns, the integration points with training and inference.
Pattern 3: Portfolio-Level Promotion Gates (7 minutes)
- Why per-model metrics can be misleading: a challenger model may outperform at the portfolio level while still underperforming on individual local level.
- Distributed backtesting with strict temporal integrity: parallel folds across thousands of entities, no leakage.
- Champion/challenger design: experimental pipelines running in parallel with production on identical folds.
- Promotion criteria: portfolio-aggregated metrics + operational robustness + an explainable driver for the improvement — not just better numbers.
- What generalizes: the aggregation logic, the promotion checklist, the parallel-pipeline architecture.
Takeaways & Q&A (3 minutes)
- The three patterns, restated as a checklist.
- The shift from a model-centric to a system-centric view of ML.
- Pointers to topics deferred to Q&A: monitoring, resilience, retraining strategy, feature store.
Machine Learning Engineer with a strong focus on building scalable and robust ML/AI platforms and systems.
I'm a Senior Data Scientist at Eneco, one of the Netherlands' leading energy companies, where I build large-scale forecasting systems that predict the energy demand and generation of millions of customers across the country. My work sits at the intersection of applied machine learning and the energy transition — turning smart meter and portfolio data into forecasts that keep the grid balanced and support the shift to renewable energy. I'm passionate about building scalable, production-grade data science solutions that solve real energy problems
I am a Machine Learning Engineer at Eneco. Always enthusiastic about designing, implementing, and scaling ML-based solutions in the energy sector to propel the transition forward!