Georg Heiler
Georg is a Senior data expert at Magenta and a ML-ops engineer at ASCII. He is solving challenges with data. His interests include geospatial graphs and time series. Georg transitions the data platform of Magenta to the cloud and is handling large scale multi-modal ML-ops challenges at ASCII.
Session
dagster-slurm is an open-source Python integration that allows data scientists and research software engineers to run Dagster pipeline assets on both a laptop and Slurm-managed HPC supercomputers without making any code changes. It automatically handles SSH transport, environment packaging via pixi-pack, and Slurm job submission, while streaming logs and scheduler metrics back to the Dagster UI in real time. The talk covers the full workflow, from local development to staging and production deployment on a real HPC cluster, using a live demo with a self-contained Docker Compose environment. It has been validated on VSC-5 in Austria and CINECA Leonardo in Italy.