2026-09-10 –, Room 2 (350)
Representation learning has transformed AI for language and vision. Large foundation models learn abstract, reusable features that can transfer to arbitrary downstream tasks. But what about other real world problems ?
Many practitioners work with inputs that are high-dimensional containing complex non-linear patterns: weather or satellite maps, sensor networks, graph data or medical scans. In contrast, prediction targets are often simple and low-dimensional. Learning directly from raw high-dimensional inputs to low-dimensional targets is tempting, but often fails in practice: limited labeled data forces models to simultaneously discover useful representations and solve the prediction task.
This talk presents the embed first, predict later paradigm and shows that it works just as powerfully for other domains as it does for text and images, using energy forecasting from weather data as our running example. We show how a pre-trained open source foundation model learns abstract representations of weather such as storms or cloud systems, and how these embeddings can be fed into a multitude of lightweight downstream tasks. The result is a modular and reusable ML pipeline that generalizes well.
You'll leave with three concrete things: (1) the building blocks of an embedding-first pipeline, illustrated through a real-world implementation, (2) practical tips and tricks for visualizing and inspecting high-dimensional embedding spaces and (3) an intuition for when this paradigm can be applied to your own domain and pointers on how to get started.
Beyond Text and Images
In computer vision, nobody hand-crafts features anymore. The field went from manual feature engineering to end-to-end CNNs to self-supervised foundation models like DINO that produce reusable representations from unlabeled data. Same story in NLP. The lesson both fields converged on: learning what things are and learning what to predict from them are better done as two separate steps.
Often people working with high-dimensional real world data beyond text and vision haven't made that jump yet. The default is still: pick some variables that you think represent your input space and relate to your target, flatten them into a dataframe, and feed it to a tree-based model or neural network. It gets you surprisingly far. But when your input space is high-dimensional and complex while targets are low-dimensional - often single numbers - this approach leaves signal on the table and tends not to generalize to related tasks.
This talk is about closing that gap and giving the audience the tools to do it themselves.
The Representation Bottleneck
The obvious alternative is to train a larger (transformer) neural network end-to-end on the raw data. But in specialized domains, labeled data is hard to come by. e.g. a model trying to predict energy generation from weather has to figure out what storms and cloud systems look like at the same time as learning how they relate to the target variable, all only guided by a scarcely available loss signal. In practice the results are often disappointing and “deep learning doesn’t work for our use case” becomes the sentiment.
Building Blocks of the Pipeline
We walk through three components that together form a reusable recipe:
1. The backbone. This can be taking the encoder of a publicly available pre-trained foundation model, in our example we are using the encoder of the Aurora AI weather model by Microsoft Research. Alternatively, a backbone can also be trained e.g. using masked autoencoders. The task of the backbone is to compress raw inputs into dense embeddings that capture abstract patterns, in our example weather regimes, seasonal cycles etc.. The backbone is trained without any task-specific supervision. We discuss how to pick a backbone that fits your domain and what to look for.
2. The embeddings. Dense numerical vector representations created by passing raw high-dimensional input (e.g. weather data) through the backbone.
3. Downstream models. Small models that map embeddings to a specific target. These models are cheap to train, e.g. using personal hardware or a single GPU. The same set of embeddings can serve many different downstream tasks. Put differently, the downstream model picks relevant representations from all representations extracted by the backbone. In our example this can be wind production forecasting, solar prediction, energy demand estimation, or insurance damage assessment. No need to rerun the backbone for each new task.
Outline
The talk introduces a general paradigm, but we spend most of the time on our concrete showcase, always circling back to how each step generalizes and how the audience can apply it to their own use case.
- Introduction and Problem Setting: 5 minutes
- Building blocks: 5 min
- Energy/Weather data showcase: 15 min
- Examples from other domains, practical tips and Q&A: 5 min
Kai Jeggle is a Meteo AI Scientist at Dexter Energy, where he turns high-dimensional weather and satellite data into insights for energy price and power generation forecasting using geospatial AI. He holds a PhD from ETH Zurich, where his research focused on combining machine learning with atmospheric physics, including time at the European Space Agency's Phi-Lab. He previously also worked as a software engineer at Hopsworks, an MLOps startup in Stockholm. Kai has been active in the AI for climate community by leading initiatives at Climate Change AI, a global non-profit working at the intersection of machine learning and climate action.