Hernan Picatto
Hernan Picatto is a Data Engineer at the Supply Chain Intelligence Institute Austria (ASCII) and a PhD student in Informatics at TU Wien. His work bridges the gap between modern data orchestration and High-Performance Computing (HPC), focusing on reproducible workflows for web-scale NLP. Hernan is a core contributor to dagster-slurm and currently manages pipelines that process petabytes of Common Crawl data to reconstruct global supply chain networks. Before returning to academia, he worked as a Senior Software Engineer at JPMorgan Chase and an Algorithm Engineer at ZhiShou Technology in Beijing.
Session
dagster-slurm is an open-source Python integration that allows data scientists and research software engineers to run Dagster pipeline assets on both a laptop and Slurm-managed HPC supercomputers without making any code changes. It automatically handles SSH transport, environment packaging via pixi-pack, and Slurm job submission, while streaming logs and scheduler metrics back to the Dagster UI in real time. The talk covers the full workflow, from local development to staging and production deployment on a real HPC cluster, using a live demo with a self-contained Docker Compose environment. It has been validated on VSC-5 in Austria and CINECA Leonardo in Italy.