2026-07-16 –, Johnson Great Room
dagster-slurm is an open-source Python integration that allows data scientists and research software engineers to run Dagster pipeline assets on both a laptop and Slurm-managed HPC supercomputers without making any code changes. It automatically handles SSH transport, environment packaging via pixi-pack, and Slurm job submission, while streaming logs and scheduler metrics back to the Dagster UI in real time. The talk covers the full workflow, from local development to staging and production deployment on a real HPC cluster, using a live demo with a self-contained Docker Compose environment. It has been validated on VSC-5 in Austria and CINECA Leonardo in Italy.
Motivation
Scientific and data engineering pipelines often span multiple compute tiers, such as preprocessing on a workstation, model training on an HPC cluster, and downstream analytics on a cloud VM. However, this is typically managed poorly, with the HPC step being a hand-written sbatch script that is disconnected from the rest of the pipeline. This results in no shared lineage, no unified observability, and no automated trigger of downstream work when the job finishes. As a result, research software engineers have to maintain two separate codebases and two mental models of the same workflow.
What dagster-slurm does
dagster-slurm is a Dagster ComputeResource and PipesClient that enables Dagster Software-Defined Assets to run on Slurm HPC clusters. You can redirect a Python function decorated with @dg.asset to a supercomputer by simply setting ExecutionMode.SLURM, without needing to make any other code changes. The library handles tasks such as SSH connection management, automatic environment packaging, Slurm job submission, and log and metadata streaming back to the Dagster UI.
Dagster-slurm is designed for teams that need an orchestrator that handles HPC. Allowing HPC workloads to be integrated into the same asset graph with full lineage, scheduling, and observability.
Talk structure (25 minutes)
The problem: why HPC and data orchestration are still separate (3 min)
Architecture overview: ComputeResource, Dagster Pipes over SSH, and pixi-pack (5 min)
Live demo: running an asset locally and then submitting it to a containerized Slurm cluster with real-time log streaming (10 min)
Lessons from production use: environment portability, air-gapped clusters, and site-specific authentication (4 min)
Roadmap and how to contribute (3 min)
The demo uses a self-contained Docker Compose stack that runs on a laptop, with no need for external cluster connectivity.
Audience and outcomes
This talk is intended for research software engineers, data engineers, and ML practitioners who work with Python pipelines and occasionally need HPC resources. Attendees will learn how to connect an existing Dagster project to a Slurm cluster and understand the design tradeoffs between task-level HPC frameworks and asset-based data orchestration.
Links: https://github.com/ascii-supply-networks/dagster-slurm | https://dagster-slurm.geoheil.com | JOSS paper (under review)
Hernan Picatto is a Data Engineer at the Supply Chain Intelligence Institute Austria (ASCII) and a PhD student in Informatics at TU Wien. His work bridges the gap between modern data orchestration and High-Performance Computing (HPC), focusing on reproducible workflows for web-scale NLP. Hernan is a core contributor to dagster-slurm and currently manages pipelines that process petabytes of Common Crawl data to reconstruct global supply chain networks. Before returning to academia, he worked as a Senior Software Engineer at JPMorgan Chase and an Algorithm Engineer at ZhiShou Technology in Beijing.
Georg is a Senior data expert at Magenta and a ML-ops engineer at ASCII. He is solving challenges with data. His interests include geospatial graphs and time series. Georg transitions the data platform of Magenta to the cloud and is handling large scale multi-modal ML-ops challenges at ASCII.