2024-08-29 –, Room 7
Foundation models are here for forecasting! This will conclusively solve all forecasting problems with a one-model-fits-all approach! Or … maybe not?
Fact is, an increasingly growing number of foundation models for time series and forecasting hitting the market.
To innocent end users, this situation raises various challenges and questions. How do I integrate the models as candidates into existing forecasting workflows? Are the models performant? How do they compare to more classical choices? Which one to pick? How to know whether to “upgrade”?
At sktime, we have tried so you don’t have to! Although you will probably be forced to anyway, but even then, it’s worth sharing experiences.
Our key challenges and findings are presented in this talk – for instance, the unexpected fragmentation of the ecosystem, difficulties in evaluating the models fairly, and more.
(sktime is an openly governed community with neutral point of view. You may be surprised to hear that this talk will not try to sell you a foundation model)
During the talk, we will expand on a number of challenges we believe end users will typically face, and how we approached them:
- API fragmentation of different foundation models. Different models use vastly different interfaces - even if weights are available on model sharing platforms, they do not come with consistent specifications. This is a substantial, underestimated challenge, and providing consistent APIs is difficult (but sktime helps!)
- Customer lock-in dynamics. Many providers would like to tie you to their model and upsell. This disincentivizes interoperability and comparability – between competitor solutions, but also with classical models, baselines, or second-to-latest generation models. But comparability and interoperability are in the interest of the end user - and can be addressed with prudent architectural decisions.
- Handling of nested software backend layers. Using a foundation model in a consistent API requires one to juggle multiple layers – model framework, deep learning backend, data backend, model marketplace, fine-tuning functionality. It is even more difficult to design a coherent software API for your production system. Learnings from sktime integrations are presented.
Finally, we will report results of tests and evaluations, including aspects of software integration, performance, and trustworthiness – exclusive at EuroSciPy 2024.
Generative Models
Expected audience expertise: Domain:none
Expected audience expertise: Python:some
Project Homepage / Git: Abstract as a tweet:navigating the plethora of forecasting foundation models - challenges, evaluations, and learnings
Public link to supporting material:
founder and core developer of sktime project.
AI researcher with experience in head/principal roles in industrial R&D and in academic faculty roles.
I completed my PhD in deep learning based time series forecasting in 2023 with the Karlsruhe Institute of Technology. In sktime, I am focusing on forecasting methods (mainly deep learning based ones) and implementing pipelines.