0.12 euroscipy-2024 2024-08-26 2024-08-30 5 00:05 https://pretalx.com Europe/Berlin Room 6 Tutorial 2024-08-26T09:00:00+02:00 09:00 01:30 This tutorial will provide an introduction to Python intended for beginners. It will notably introduce the following aspects: - built-in types - controls flow (i.e. conditions, loops, etc.) - built-in functions - basic Python class euroscipy-2024-53588-introduction-to-python Community, Education, and Outreach Mojdeh Rastgoo en This tutorial will provide an introduction to Python intended for beginners. It will notably introduce the following aspects: - built-in types - controls flow (i.e. conditions, loops, etc.) - built-in functions - basic Python class We introduce here the Python language. You can find the material for the tutorial here: https://github.com/mrastgoo/python-introduction-jupyterlite You can either clone the repository and run the tutorial on your laptop. Or use the following link : https://mrastgoo.github.io/python-introduction-jupyterlite/ to run the tutorial in jupyterlite. false https://pretalx.com/euroscipy-2024/talk/Z98TJA/ https://pretalx.com/euroscipy-2024/talk/Z98TJA/feedback/ Room 6 Tutorial 2024-08-26T11:00:00+02:00 11:00 01:30 Are you starting to use Python for scientific computing? Join this tutorial to know more about NumPy, the building block for nearly all libraries in the scientific ecosystem. You will learn how to manipulate NumPy arrays, understand how they store data and discover how to get optimal performances. By the end of this tutorial, you will be able to start working with NumPy and know the main pitfalls to avoid. euroscipy-2024-52823-introduction-to-numpy Data Science and Visualisation Sarah Diot-Girard en This is a hands-on workshop, please bring a laptop. You can find the installation instructions for the tutorial here: https://github.com/SdgJlbl/numpy-introduction-tutorial#installation-instructions. A back-up online solution will be available if you are not able to install everything locally. Target audience: beginner in the Python scientific ecosystem, some basic knowledge of Python and its tooling are a plus. Agenda: - What is NumPy and when to use it? - 10 min - Workshop set-up - 5 min - Creating and manipulating NumPy arrays - 10 min - Basic indexing - 10 min - Shape and broadcasting - 20 min - Filtering and masking - 15 min - Vectorized operations - 15 min - Wrap-up and key take-away - 5 min false https://pretalx.com/euroscipy-2024/talk/UDVD77/ https://pretalx.com/euroscipy-2024/talk/UDVD77/feedback/ Room 6 Tutorial 2024-08-26T14:00:00+02:00 14:00 01:30 _matplotlib_ is a library for creating visualizations with Python which "...makes easy things easy and hard things possible" (https://matplotlib.org/). This tutorial, intended for beginners, will introduce the library and explain core concepts as well as the main interfaces. Starting with styling simple point data plots, we will explain how to work with several dimensions, shared axes and advanced styling options using _rcParams_. After completing this tutorial, participants will hopefully be equipped with a thorough understanding of _matplotlib_ to navigate the "hard things" in the world of data visualization. euroscipy-2024-53589-introduction-to-matplotlib-for-data-visualization-with-python Data Science and Visualisation Umut Nefta Kanilmaz en _matplotlib_ is one of the most-used and powerful visualization libraries for Python which comes with a huge feature set and configuration flexibility for users. However, these traits also introduce a good amount of complexity that can be hard to tackle alone. This tutorial aims to help beginners take the inital stage and speed up learning data visualization with _matplotlib_. We will follow along this rough outline: Part 1 // The Basics or How to Talk to _matplotlib_ (45 min) - The matplotlib interfaces: _plt_ vs _fig, ax_ - Visualizing and styling two dimensional data: point and line plots, making titles, labelling axes, choosing colors, ... - Hands-on examples Part 2 // Carrying On the Conversation: More Dimensions and Figures (45 min) - Custom styling with _rcParams_ - Creating multiple figures and sharing axes - Other dimensions: working with three-dimensional and polar data - Hands-on examples false https://pretalx.com/euroscipy-2024/talk/8NL9R3/ https://pretalx.com/euroscipy-2024/talk/8NL9R3/feedback/ Room 6 Tutorial 2024-08-26T16:00:00+02:00 16:00 01:30 Scientists are producing more and more images with telescopes, microscopes, MRI scanners, etc. They need automatable tools to measure what they've imaged and help them turn these images into knowledge. This tutorial covers the fundamentals of algorithmic image analysis, starting with how to think of images as NumPy arrays, moving on to basic image filtering, and finishing with a complete workflow: segmenting a 3D image into regions and making measurements on those regions. euroscipy-2024-52993-image-analysis-in-python-with-scikit-image Data Science and Visualisation Deleted UserMarianne CorvellecStéfan van der Walt en This tutorial is aimed at folks who have some experience in scientific computing with Python, but are new to image analysis. We will introduce the fundamentals of working with images in scientific Python. At every step, we will visualize and understand our work using Matplotlib. The tutorial will be split into three parts, of about 30 minutes each: - **Images are just NumPy arrays.** In this section we will cover the basics: how to think of images not as things we can see but numbers we can analyze. - **Changing the structure of images with image filtering.** In this section we will define *filtering*, a fundamental operation on signals (1D), images (2D), and higher-dimensional images (3D+). We will use filtering to find various structures in images, such as *blobs* and *edges*. - **Finding regions in images and measuring their properties.** In this section we will define *image segmentation* — splitting up images into regions. We will show how segmentation is commonly represented in the scientific Python ecosystem, some basic and advanced methods to do it, and use it to make object measurements. false Tutorial sources https://pretalx.com/euroscipy-2024/talk/ZVBAKK/ https://pretalx.com/euroscipy-2024/talk/ZVBAKK/feedback/ Room 5 Tutorial 2024-08-26T09:00:00+02:00 09:00 01:30 Welcome to this tutorial on Python's magic methods, often underestimated or overlooked in programming practice. And yet, magic methods make it easy to write code that implements often very complex algorithms in a readable way. In this tutorial, we will learn how to create and use magic methods and talk about some interesting non-trivial magic methods. This tutorial will have classroom exercises, post class homeworks as well as complimentary readings. All the presentation, code, exercises will be shared in advance (~ 2 - 3 days) and the solutions of the exercise will be shared after the tutorial is completed. euroscipy-2024-54597-what-is-the-magic-of-magic-methods-in-the-python-language Community, Education, and Outreach Paweł Żal en Python allows you to equip created classes with special methods, also known as magic methods or dunder methods. To recognize a special method, you should know that it is a method whose name begins and ends with a double underscore. But it's not the name that their magic lies in, because these methods have a special meaning for Python. Python calls magic methods in response to fundamental operations, such as creating class instances, indexing sequences, comparing objects, managing attribute access, and more, so knowing how to create them is fundamental for any Pythonista. During this tutorial, you’ll: * Find out what Python magic methods are, * Understand the magic behind Python magic methods, * Customize various behaviors of classes using magic method. To fully benefit from this tutorial, you should be familiar with object-oriented programming in Python. false https://pretalx.com/euroscipy-2024/talk/WZQXUY/ https://pretalx.com/euroscipy-2024/talk/WZQXUY/feedback/ Room 5 Tutorial 2024-08-26T11:00:00+02:00 11:00 01:30 Python offers decorator to implement re-usable code for cross-cutting task. The support the separation of cross-cutting concerns such as logging, caching, or checking of permissions. This can improve code modularity and maintainability. This tutorial is an in-depth introduction to decorators. It covers the usage of decorators and how to implement simple and more advanced decorators. Use cases demonstrate how to work with decorators. In addition to showing how functions can use closures to create decorators, the tutorial introduces callable class instance as alternative. Class decorators can solve problems that use be to be tasks for metaclasses. The tutorial provides uses cases for class decorators. While the focus is on best practices and practical applications, the tutorial also provides deeper insight into how Python works behind the scene. After the tutorial participants will feel comfortable with functions that take functions and return new functions. euroscipy-2024-52781-decorators-a-deep-dive Scientific Applications Mike Müller en Python offers decorator to implement re-usable code for cross-cutting task. The support the separation of cross-cutting concerns such as logging, caching, or checking of permissions. This can improve code modularity and maintainability. This tutorial is an in-depth introduction to decorators. It covers the usage of decorators and how to implement simple and more advanced decorators. Use cases demonstrate how to work with decorators. In addition to showing how functions can use closures to create decorators, the tutorial introduces callable class instance as alternative. Class decorators can solve problems that use be to be tasks for metaclasses. The tutorial provides uses cases for class decorators. While the focus is on best practices and practical applications, the tutorial also provides deeper insight into how Python works behind the scene. After the tutorial participants will feel comfortable with functions that take functions and return new functions ## Audience This tutorial is for intermediate Python programmers who want to dive deeper. Solid working knowledge of functions and classes basics is required. ## Outline * Examples of using decorators * from the standard library * from third-party packages * Closures for decorators * Write a simple decorator * Best Practice * Use case: Caching * Use case: Logging * Parameterizing decorators * Chaining decorators * Callable instances instead of functions * Use case: Argument Checking * Use case: Registration * Class decorators * Wrap-up and questions false https://pretalx.com/euroscipy-2024/talk/BCAUKU/ https://pretalx.com/euroscipy-2024/talk/BCAUKU/feedback/ Room 5 Tutorial 2024-08-26T14:00:00+02:00 14:00 01:30 Data scientists are repeatedly told that it is absolutely critical to align their model training methodology with a specific business objective. While being a rather good advice, it usually falls short on details on how to achieve this in practice. This hands-on tutorial aims to introduce helpful theoretical concepts and concrete software tools to help them bridge this gap. This method will be illustrated on a worked practical use case: optimizing the operations of a fraud detection system for a payment processing platform. More specifically, we will introduce the concepts of calibrated probabilistic classifiers, how to evaluate them and fix common causes of mis-calibration. In a second part, we will explore how to turn probabilistic classifiers into optimal business decision makers. The tutorial material is available at the following URL: https://github.com/probabl-ai/calibration-cost-sensitive-learning euroscipy-2024-51444-probabilistic-classification-and-cost-sensitive-learning-with-scikit-learn Machine and Deep Learning Guillaume LemaitreOlivier Grisel en Detailed outline of the tutorial: - Introduction - Evaluting ML based predictions with: - ranking metrics, - probabilistic metrics, - decision metrics. - Proper scoring losses and their decomposition in: - calibration loss, - grouping loss, - irreducible loss. - Part I: Probabilistic classification - The calibration curve - Possible causes of miscalibration - Model misspecification - Overfitting and bad level of regularization - Possible ways to improve calibration - Non-linear feature engineering to avoid misspecification - Post-hoc calibration with Isotonic regression - Tuning parameters and early stopping with a proper-scoring rule - Part II: Optimal decision making under uncertainty - Defining a custom business cost functions - Individual-specific cost functions - Setting the Elkan-optimal threshold with `FixedThresholdClassifier` - Cost-sensitive learning for arbitrary cost functions with `TunedThresholdClassifierCV` - Predict-time decision threshold optimization. This tutorial will be delivered as a set of publicly available Jupyter notebooks under an open source license. We will mostly use components of the latest version of the scikit-learn library + a few custom extensions. The tutorial material is available at the following URL: https://github.com/probabl-ai/calibration-cost-sensitive-learning false https://pretalx.com/euroscipy-2024/talk/UNYV7V/ https://pretalx.com/euroscipy-2024/talk/UNYV7V/feedback/ Room 5 Tutorial 2024-08-26T16:00:00+02:00 16:00 01:30 Python code that works with Numpy, Cupy and PyTorch arrays? Use a GPU when possible, but fallback to using a CPU if there is none? We will show you how you can write Python code that can do all of the above. The not so secret ingredient to do this is the Array API. In this workshop you will learn what the Array API is and how to use it to write programs that can take any compatible array as input. euroscipy-2024-50877-using-the-array-api-to-write-code-that-runs-with-numpy-cupy-and-pytorch High Performance Computing Tim HeadSebastian Berg en There are many Python libraries to choose from for numerical computing, data science, machine learning and deep learning. A downside of this diversity is that the API of each of these array libraries is subtly different. This makes it hard to write code that works with more than one array type. As a result taking advantage of modern hardware, like a GPU, is hard because you need to handle the differences between Numpy and an array library that supports GPUs. The Array API standard aims to solve this problem by providing a common API that all compatible libraries support. This means that you can write code that works no matter what array library is used. And, because there are array libraries with GPU support, it means you can write Python code that works on CPUs and GPUs! This workshop will be hands on! This means you need to bring a laptop that has at least Numpy and PyTorch installed on it. This will let you experience that your code works with either of these libraries. To see the effect of using a GPU you will also need to either have a laptop that has one or use a service like Google colab. After a brief introduction and demo we will tackle one or two applications that can be implemented in Numpy and then modified to use the Array API. By the end of this workshop you will know how to take Numpy code and modify it so that it is compatible with the Array API. You will be well equipped to modify existing libraries in the PyData ecosystem or write your own applications. **Material: https://github.com/betatim/sound-array-api-tutorial** false Material https://pretalx.com/euroscipy-2024/talk/89KK7L/ https://pretalx.com/euroscipy-2024/talk/89KK7L/feedback/ Room 6 Tutorial 2024-08-27T09:00:00+02:00 09:00 01:30 Polars is a new, powerful library for doing analysis on structured data. The library focuses on processing speed and a consistent and intuitive API. This tutorial will help you get started with Polars, by showing you how to read and write data and manipulate it with Polars' powerful expression syntax. You'll learn about how the lazy API is an important key to Polars' efficiency. euroscipy-2024-51614-introduction-to-polars-fast-and-readable-data-analysis Data Science and Visualisation Geir Arne Hjelle en Polars is a new, lightning-fast library for analyzing structured data. The library focuses on processing speed and a consistent and intuitive API. Its syntax supports transformations like selection, filtering, and aggregation with dedicated and powerful expressions. Polars does lazy evaluation out-of-the-box with an advanced query planner. In this tutorial, you'll learn how you can manipulate your data with Polars. You'll start by reading existing data into a Polars DataFrame and learn how to use _tidy_ principles to organize your analysis workflow. After learning the basics of Polars, you'll start exploring Polars' lazy interface, which is where the library really shines. With the lazy API, queries are only executed when the results are needed. This can improve performance significantly, as Polars can take advantage of several different optimizations. Throughout the tutorial, you'll gain experience working lazily. You'll learn how to inspect the optimized query plan and how to play to the library's strengths. This tutorial is for anyone curious about Polars. You don't need previous experience with other libraries like pandas, but if you have used pandas earlier, you'll learn how Polars is different and how the libraries can play nicely together. false https://pretalx.com/euroscipy-2024/talk/MPMRUZ/ https://pretalx.com/euroscipy-2024/talk/MPMRUZ/feedback/ Room 6 Tutorial 2024-08-27T11:00:00+02:00 11:00 01:30 Learning NLP often requires a corpus of sample texts. The common choice is Wikipedia. The project is open source and has huge amounts of a natural language content in dozens of languages. Happily the Wikimedia Foundation publishes data dumps in XML format, which could be easily parsed. In this tutorial you will learn how to do that in Python. euroscipy-2024-54986-using-wikipedia-as-a-language-corpus-for-nlp Scientific Applications Jakub B. Jagiełło en In this tutorial you will learn where to find the Wikipedia dumps, how to use Python’s built-in XML parser together with a MediaWiki syntax parser (mwparserfromhell) to extract raw text from Wikipedia articles. We will also discuss the difference between streaming and in-memory parsers, and why the former are better for parsing huge amounts of data. We will discuss the typical NLP stream, and as an example of additional steps needed in inflected languages, we will use a morphological analyser to lemmatise words sourced from polish language Wikipedia to calculate their frequencies. As an example application we will compare such statistics with a Polish language corpus available in Python’s NLTK library (“pl196x” module, with the IPI PAN corpus of polish language of the 1960s) and show lexical differences between both corpora. false https://pretalx.com/euroscipy-2024/talk/XZVGDB/ https://pretalx.com/euroscipy-2024/talk/XZVGDB/feedback/ Room 6 Tutorial 2024-08-27T14:00:00+02:00 14:00 01:30 With Machine Learning becoming a topic of high interest in the scientific community, over the years, many different programming languages and environments have been used for Machine Learning research and system development. Python is known as easy to learn, yet powerful programming languages and has become a popular choice among professionals and amateurs. This tutorial will provide instructions on the usage of two popular Python libraries: Scikit-learn and Pandas, in Machine Learning modeling. euroscipy-2024-54282-introduction-to-machine-learning-with-scikit-learn-and-pandas Machine and Deep Learning Justyna Szydłowska-Samsel en With Machine Learning becoming a topic of high interest in the scientific community, over the years, many different programming languages and environments have been used for Machine Learning research and system development. Python is known as easy to learn, yet powerful programming languages and has become a popular choice among professionals and amateurs. This tutorial will provide instructions on the usage of two popular Python libraries: Scikit-learn and Pandas, in Machine Learning modeling. The tutorial includes: - data preparation for ML modeling - introduction of basic ML models - implementation of a basic ML model false Github repository https://pretalx.com/euroscipy-2024/talk/WVZPXM/ https://pretalx.com/euroscipy-2024/talk/WVZPXM/feedback/ Room 6 Tutorial 2024-08-27T16:00:00+02:00 16:00 01:30 Open-source projects are essential for scientific programming. They provide many tools and resources that can be customized for different scientific needs. However, sometimes the existing tools in a package don't meet all the requirements of a project. This is when contributing to open-source packages becomes important. By contributing, you can implement new functionalities, improve the software and help keep the open-source community strong. This workshop will make contributing to open-source projects easier to understand. It will guide participants from just using the software to actively contributing to it. The workshop will address technical challenges such as interacting with web-based hosting services (like GitHub and GitLab), branching, and opening pull requests. Additionally, it will cover how to contribute documentation and ensure the correctness of the code. We will use the following repository during the workshop: https://github.com/Nikoleta-v3/HitchCos. You can find a checklist of prerequisites and installation notes here: https://github.com/Nikoleta-v3/HitchCos/wiki/Prerequisites. euroscipy-2024-51602-a-hitchhiker-s-guide-to-contributing-to-open-source Community, Education, and Outreach Sebastian BergNikoleta E. Glynatsi en In this tutorial, we will introduce the process of contributing to open source projects. The workshop will begin by covering basic Git commands, and we will demonstrate how to interact with GitHub, fork and clone repositories. Moreover, we will discuss how to implement functions in a Pythonic way, how to document and test them. We will also cover topics such as online documentation and Continuous Integration (CI) systems. For each part, there will be associated exercises that will ultimately lead participants to contribute to a repository we have set up for the workshop. Participants can choose what to work on from a list of issues. These issues will include implementing functions, testing current functions, documenting current functions, and correcting typos in online documentation material. Finally, we will discuss and demonstrate, through the online repository we will have set up, how to interact with the open-source community, look for guidelines to follow styles, and testing guidelines. To also familiarize participants with the process of reviewing and commenting, we will comment on and provide feedback to the pull requests opened by the participants during the workshop as well as in the following days. false https://pretalx.com/euroscipy-2024/talk/UF7LM8/ https://pretalx.com/euroscipy-2024/talk/UF7LM8/feedback/ Room 5 Tutorial 2024-08-27T09:00:00+02:00 09:00 01:30 In computational science, different software packages are often glued together as scripts to perform numerical experiments. With increasing complexity, these scripts become unmaintainable, prone to crashes, hard to scale up and to collaborate on. AiiDA solves these problems via a powerful workflow engine and by keeping provenance for the entire workflow. In this tutorial, we learn how to create dynamic workflows combining together different executables that automatically can restart from failed runs and reuse results from completed calculations via caching. euroscipy-2024-50826-building-robust-workflows-with-strong-provenance Scientific Applications Alexander GoscinskiJulian GeigerAli Khosravi en Have you ever built a computational script for running calculations and lost track of the data you produced? Have you submitted your script to a high-performance cluster (HPC) and your job failed so you needed to restart the whole workflow? Did you want to streamline the production and access of computational experiment results? By writing your workflow in AiiDA, intermediate and final results are stored in a structured manner in a database. In addition, you can restart from the last checkpoint and reuse results from duplicated calculations via caching. As such, AiiDA not only helps you with your personal data management, but also enables easy sharing with other collaborators. This is a hands on session which is structured in the following way: Part 1: Introduction to AiiDA - what problems can it help you to solve (20 mins) - Provenance, a robust solution for process management and data traceability - Scalability, interoperability, and high-throughput performance Part 2: How to quickly create a workflow from a set of executables (40 mins) - Quickly set up a running instance - Concatenating several scripts to one workflow - Parsing output files to filter out meaningful results from outputs Part 3: How to create more complex workflows (30 mins) - Implementing concurrent jobs in graph-like dependencies - Generate on-the-fly a workflow from input - Querying results from the AiiDA database By the end of the tutorial, you will have learned how to use AiiDA to quickly create workflows that leverage its restart and caching capabilities. You will learn how to implement workflows with graph-like dependencies to run their calculations concurrently, and how to access and share their results. You can follow this tutorial by using the development environment provided by https://nanohub.org/tools/aiida.. Because nanohub changes the path when making the environment publically available, you need to run the following command in one of the jupyter cells to run notebook 2 and 3 ``` !echo "export PATH=$PATH:$(realpath ../../data/euro-scipy-2024/diag-wf):$(realpath ../../data/euro-scipy-2024/diag-wf/bin/default)" >> ~/.bash_profile ``` The support thread for the tutorial on Discourse can be found at the following link: https://aiida.discourse.group/t/euroscipy-2024-support/456 false Development environment to follow tutorial https://pretalx.com/euroscipy-2024/talk/BYESWT/ https://pretalx.com/euroscipy-2024/talk/BYESWT/feedback/ Room 5 Tutorial 2024-08-27T11:00:00+02:00 11:00 01:30 Polars is a dataframe library taking the world by storm. It is very runtime and memory efficient and comes with a clean and expressive API. Sometimes, however, the built-in API isn't enough. And that's where its killer feature comes in: plugins. You can extend Polars, and solve practically any problem. No prior Rust experience required, intermediate Python or general programming experience required. By the end of the session, you will know how to write your own Polars Plugin! This talk is aimed at data practitioners. euroscipy-2024-52971-combining-python-and-rust-to-create-polars-plugins Data Science and Visualisation Marco Gorelli en Have you ever had the experience of needing to write a really custom function? Did you end up using a custom Python lambda function and waiting endlessly whilst your code executed? Learn how to put an end to that! This tutorial is aimed at advanced dataframe users who want to go beyond what Polars offers them. The structure will be: - 5 minutes motivation: example of a custom function which is painfully slow - 30 minutes: the bare minimum Rust you need to know in order to write a Polars plugin - 20 minutes: let's get something running! Starting from a cookiecutter template, let's glue pieces together and get a simple "pig-latinnifier" running - 25 minutes: customising the basic "pig-latinnifier" to implement that same custom function as a plugin - 5 minutes: let's glue things together, run the plugin, and observe how much faster it is! - 5 minutes: assorted requests / Q&A This may look ambitious - however, I have taught Polars Plugins professionally and have given talks about the topic before, so I'm confident that it is doable. By the end of the session, attendees will know how to write their own Polars Plugin. This talk is aimed at data practitioners who have experience with Python and data analysis (however, no prior Rust experience is required!). If you want to follow the tutorial on your own laptop, then you will need to come prepared with the following installed: - Rust (see https://rustup.rs/) - an IDE, ideally with the Rust Analyzer installed - a Python3.9+ virtual environment, in which you should install Polars and Maturin If you can follow the instructions at https://github.com/MarcoGorelli/cookiecutter-polars-plugins, you'll be off to a flying start! false https://pretalx.com/euroscipy-2024/talk/7SKUEN/ https://pretalx.com/euroscipy-2024/talk/7SKUEN/feedback/ Room 5 Tutorial 2024-08-27T14:00:00+02:00 14:00 01:30 Inspired by xarray, Scipp enriches raw NumPy-like multi-dimensional data arrays by adding named dimensions and associated coordinates. For an even more intuitive and less error-prone user experience, Scipp adds physical units to arrays and their coordinates. Through this tutorial, participants will learn about the basics of modelling their data with the Scipp library and using in built tools in Scipp for scientific data analysis. One of Scipp's key features is the possibility of using multi-dimensional non-destructive binning to sort record-based "tabular"/"event" data into arrays of bins. This provides fast and flexible binning, rebinning, and filtering operations, all while preserving the original individual records. Scipp ships with data display and visualization features for Jupyter notebooks, including a powerful plotting interface. Named Plopp, this tool uses a graph of connected nodes to provide interactivity between multiple plots and widgets, requiring only a few lines of code from the user. Scipp is available via pip and conda and runs on Linux, Mac and Windows. euroscipy-2024-50917-multi-dimensional-arrays-with-scipp High Performance Computing Mridul Seth en In this tutorial we will cover various key features of Scipp, a python library with a C++ core. During the tutorial we will go through multiple notebooks with hands on exercises. Part A (30 mins) - Introduction to general concepts in scipp - Basic data structures in scipp Part B (30 mins) - Binning of data and computation on top of it. - Tips and tricks of data analysis on top of multi dimensional arrays. Part C (30 mins) - Visualizing scipp data with plopp. - Interop with the wider scientific python ecosystem. - File I/O. By the end of the tutorial we hope that participants will be comfortable with using the scipp API to model and analyze their data. false https://pretalx.com/euroscipy-2024/talk/8WL8GX/ https://pretalx.com/euroscipy-2024/talk/8WL8GX/feedback/ Room 5 Tutorial 2024-08-27T16:00:00+02:00 16:00 01:30 sktime is the most widely used scikit-learn compatible framework library for learning with time series. sktime is maintained by a neutral non-profit under permissive license, easily extensible by anyone, and interoperable with the python data science stack. This tutorial gives a hands-on introduction to sktime, for common time series learning tasks such as forecasting, starting with a general overview of the package and forecasting interfaces for uni- and multivariate forecasts with endo-/exogeneous data, probabilistic forecasts, and forecasting in the presence of hierarchical data. The tutorial then proceeds to showcase some of the newest features in 2024, based on a hierarchical demand forecasting use case example: support for foundation models, hugging face connectors, advanced support for hierarchical and global forecasts, and integration features for creating API compatible algorithms and sharing them via the sktime discoverability tools. euroscipy-2024-50919-sktime-python-toolbox-for-time-series-introduction-and-new-features-2024-foundation-models-deep-learning-backends-probabilistic-models-hierarchical-demand-forecasting-marketplace-features Machine and Deep Learning Franz KiralyFelipe AngelimMuhammad Armaghan ShakirBenedikt Heidrich en The tutorial gives an up-to-date introduction to sktime base features with a focus on forecasting, model building, hierarchical and global data, and marketplace features. It showcases a selection of new and exciting features 2024: - Integrations for foundation models, pre-trained or fine-tuned deep learning models, hugging face connector - global forecasting interfaces, building parallelizable pipelines for hierarchical data sets with level individual models and autoML - Probabilistic models, distribution prediction, reduction to tabular probabilistic regression - New developer marketplace patterns for developing and registering API compatible estimators with the sktime estimator search and discoverability tools sktime is developed by an open community, with aims of ecosystem integration in a neutral, charitable space. We welcome contributions and seek to provides opportunity for anyone worldwide. false https://pretalx.com/euroscipy-2024/talk/DF3VHU/ https://pretalx.com/euroscipy-2024/talk/DF3VHU/feedback/ Room 7 Keynote 2024-08-28T09:00:00+02:00 09:00 01:00 A lot has been happening in the field of AI and Natural Language Processing: there's endless excitement about new technologies, sobering post-hype hangovers and also uncertainty about where the field is heading next. In this talk, I'll share the most important lessons we've learned in 10 years of working on open source software, our core philosophies that helped us adapt to an ever-changing AI landscape and why open source and interoperability still wins over black-box, proprietary APIs euroscipy-2024-53390-10-years-of-open-source-navigating-the-next-ai-revolution Community, Education, and Outreach Ines Montani en A lot has been happening in the field of AI and Natural Language Processing: there's endless excitement about new technologies, sobering post-hype hangovers and also uncertainty about where the field is heading next. In this talk, I'll share the most important lessons we've learned in 10 years of working on open source software, our core philosophies that helped us adapt to an ever-changing AI landscape and why open source and interoperability still wins over black-box, proprietary APIs false https://pretalx.com/euroscipy-2024/talk/JFATCJ/ https://pretalx.com/euroscipy-2024/talk/JFATCJ/feedback/ Room 7 Talk (25 mins + Q&A) 2024-08-28T10:30:00+02:00 10:30 00:30 In this talk, we'll review the landscape of open-source federated learning libraries with a lens on actual real world data problems, use cases and actors who could benefit from federated learning. We'll then analyze gaps, weaknesses and explore new ways we could formulate federated learning problems (and their associated libraries!) to build more useful software and use decentralized machine learning in real world use cases. euroscipy-2024-50694-federated-learning-where-we-are-and-where-we-need-to-be Machine and Deep Learning Katharine Jarmul en This talk will introduce: - Aspects of federated learning that are important for real world use cases - Federated learning libraries available via open-source - An evaluation of federated learning open-source libraries - A gap analysis of potential problems when leveraging open-source for real world use cases - Suggestions for navigating this gap and building supporting libraries or new open-source solutions to address the discovered problems false https://pretalx.com/euroscipy-2024/talk/PFVX9L/ https://pretalx.com/euroscipy-2024/talk/PFVX9L/feedback/ Room 7 Talk (25 mins + Q&A) 2024-08-28T11:05:00+02:00 11:05 00:30 Helmholtz Blablador is the LLM inference server from the Helmholtz Foundation. This talk explores Blablador's role in hosting open-source LLM models and models developed in-house at the Juelich Supercomputing Centre (JSC). This talk is about Blablador and the open source LLM models' ecosystem. euroscipy-2024-51075-helmholtz-blablador-and-the-llm-models-ecosystem Machine and Deep Learning Alexandre Strube en In the ever-evolving world of machine learning, the Helmholtz Foundation's Blablador stands out as an open LLM inference server/service for the academic community. This talk discusses Blablador and its role in hosting both open-source LLM models, models developed by the academic community in general, and those developed in-house at the Juelich Supercomputing Centre (JSC). Blablador not only supports a wide range of models but also provides a robust platform for researchers and developers to experiment, collaborate, and innovate. As a result, Blablador has significantly contributed to the growth and advancement of the LLM models' ecosystem. This talk will look at the architecture and functionality of Blablador, its integration with the JSC, and its impact on the LLM models' ecosystem. We explore Blablador's role in fostering collaboration and innovation in the machine learning community. false https://pretalx.com/euroscipy-2024/talk/UXHSQC/ https://pretalx.com/euroscipy-2024/talk/UXHSQC/feedback/ Room 7 Talk (15 mins + Q&A) 2024-08-28T11:40:00+02:00 11:40 00:20 Scikit-LLM is an innovative Python library, seamlessly integrates Large Language Models into the Scikit-Learn framework. Scikit-LLM becomes a powerful tool for natural language processing (NLP) tasks within the Scikit-Learn pipeline, and I'll showcase a data augmentation action to build features using zero-shot text classification and text vectorization. euroscipy-2024-50915-data-augmentation-with-scikit-llm Machine and Deep Learning Claudio Giorgio Giancaterino en Scikit-learn is one of the most well-known and widely-used open-source Python libraries in the field of machine learning by data scientists due to its wide range of models and friendly use. You are able to solve any task, from regression to classification, from clustering to dimensionality reduction, using just one library. Scikit-LLM is a Python library that embodies large language models into the scikit-learn framework. It’s a tool to perform natural language processing (NLP) tasks all within the Scikit-Learn pipeline. The features provided by Scikit-LLM are -Zero-Shot Text Classification -Few-Shot Text Classification -Dynamic Few-Shot Text Classification -Multi-Label Zero-Shot Text Classification -Text Vectorization -Text Translation -Text Summarization. Will be presented an use case of data augmentation for flood event from the US storm events database using zero-shot text classification and embeddings techniques. false https://pretalx.com/euroscipy-2024/talk/CBSXQN/ https://pretalx.com/euroscipy-2024/talk/CBSXQN/feedback/ Room 7 Talk (25 mins + Q&A) 2024-08-28T13:20:00+02:00 13:20 00:30 When it comes to designing machine learning predictive models, it is reported that data scientists spend over 80% of their time preparing the data to input to the machine learning algorithm. Currently, no automated solution exists to address this problem. However, the `skrub` Python library is here to alleviate some of the daily tasks of data scientists and offer an integration with the `scikit-learn` machine learning library. In this talk, we provide an overview of the features available in `skrub`. First, we focus on the preprocessing stage closest to the data sources. While predictive models usually expect a single design matrix and a target vector (or matrix), in practice, it is common that data are available from different data tables. It is also possible that the data to be merged are slightly different, making it difficult to join them. We will present the `skrub` joiners that handle such use cases and are fully compatible with `scikit-learn` and its pipeline. Then, another issue widely tackled by data scientists is dealing with heterogeneous data types (e.g., dates, categorical, numerical). We will present the `TableVectorizer`, a preprocessor that automatically handles different types of encoding and transformation, reducing the amount of boilerplate code to write when designing predictive models with `scikit-learn`. Like the joiner, this transformer is fully compatible with `scikit-learn`. euroscipy-2024-51594-skrub-prepping-tables-for-machine-learning Machine and Deep Learning Guillaume LemaitreVincent MaladiereJérôme Dockès en When it comes to designing machine learning predictive models, it is reported that data scientists spend over 80% of their time preparing the data to input to the machine learning algorithm. Currently, no automated solution exists to address this problem. However, the `skrub` Python library is here to alleviate some of the daily tasks of data scientists and offer an integration with the `scikit-learn` machine learning library. In this talk, we provide an overview of the features available in `skrub`. First, we focus on the preprocessing stage closest to the data sources. While predictive models usually expect a single design matrix and a target vector (or matrix), in practice, it is common that data are available from different data tables. It is also possible that the data to be merged are slightly different, making it difficult to join them. We will present the `skrub` joiners that handle such use cases and are fully compatible with `scikit-learn` and its pipeline. Then, another issue widely tackled by data scientists is dealing with heterogeneous data types (e.g., dates, categorical, numerical). We will present the `TableVectorizer`, a preprocessor that automatically handles different types of encoding and transformation, reducing the amount of boilerplate code to write when designing predictive models with `scikit-learn`. Like the joiner, this transformer is fully compatible with `scikit-learn`. false https://pretalx.com/euroscipy-2024/talk/MFF7GE/ https://pretalx.com/euroscipy-2024/talk/MFF7GE/feedback/ Room 7 Talk (15 mins + Q&A) 2024-08-28T13:55:00+02:00 13:55 00:20 The availability of AI models and packages in the Python ecosystem has revolutionized many applications across domains. This talk discusses infrastructural decisions and best practices that bridge the gap between interactive data analyses in notebooks and production applications at a reasonable scale, suitable for both commercial and scientific contexts. In particular, the talk introduces the on-premises, Python-based AI architecture employed at MDPI, one of the largest open-access publishers. The presentation emphasizes the impact of the design on reproducibility, decoupling of different resources, and ease of use during the development and exploration phases. euroscipy-2024-50831-from-data-analysis-in-jupyter-notebooks-to-production-applications-ai-infrastructure-at-reasonable-scale Machine and Deep Learning Frank Sauerburger en While there is certainly no shortage of tutorials on how to build AI applications in a Jupyter notebook, it can be challenging to move from proof-of-concepts to reliable and reproducible data analyses used for data-driven decisions, or production-grade applications. The presentation discusses architectural decisions in a Python-based environment to bridge this gap at typical scales in academia and industry. Splitting the system into smaller composable building blocks provides reproducibility, more rapid development, and more efficient use of available resources, and has enabled MDPI to leverage AI at multiple stages of the business process. The concepts presented in the talk apply to a wide range of applications. false https://pretalx.com/euroscipy-2024/talk/8NJGVH/ https://pretalx.com/euroscipy-2024/talk/8NJGVH/feedback/ Room 7 Talk (15 mins + Q&A) 2024-08-28T14:25:00+02:00 14:25 00:20 This presentation introduces a Qdrant vector DB and Specter2 model used to identify whether a rejected academic manuscript is later published in a competing journal. Our method combines AI, data science and analytics to ensure a good identification of manuscripts and authors. The findings offer insights into resubmission patterns, enhancing our understanding of academic publishing dynamics. The system is implemented in Python. euroscipy-2024-50830-a-qdrant-and-specter2-framework-for-tracking-resubmissions-of-rejected-manuscripts-in-academia Data Science and Visualisation Daniele Raimondi en Understanding what happens to rejected manuscripts is crucial in academic publishing. We developed a system to track whether rejected manuscripts are later published in competing journals using advanced machine-learning techniques. By extracting rejected manuscript embeddings with the Specter2 model and storing them in a vector database, we compare these with published articles, focusing on title and abstract similarities. Author similarity and other checks ensure accurate identification despite author name variations. Our system generates two key scores: manuscript similarity and author similarity. A machine learning approach classifies papers as the same or different, with thresholds fine-tuned through manual labelling and scatter plot analysis. This approach combines AI, data science and analytics, providing valuable insights into resubmission patterns and enhancing our understanding of academic publishing dynamics. false https://pretalx.com/euroscipy-2024/talk/HCMV78/ https://pretalx.com/euroscipy-2024/talk/HCMV78/feedback/ Room 6 Talk (25 mins + Q&A) 2024-08-28T10:30:00+02:00 10:30 00:30 Many scientific Python packages are "stringly typed," i.e., using strings to select algorithms or methods and dictionaries for configuration. While easy for beginners and convenient for authors, these libraries miss out on static typing benefits like error detection before runtime and autocomplete. This talk shares insights from redesigning the optimagic library from the ground up with static typing in mind. Without compromising on simplicity, we achieve better static analysis, autocomplete, and fewer runtime errors. The insights are not specific to numerical optimization and apply to a wide range of scientific Python packages. euroscipy-2024-50658-from-stringly-typed-to-strongly-typed-insights-from-re-designing-a-library-to-get-the-most-out-of-type-hints Community, Education, and Outreach Janos Gabler en Most libraries in the scientific Python ecosystem are "stringly typed". For example, many packages allow users to switch between multiple algorithms or methods by passing a string with the method name; Advanced configuration often relies on dictionaries of keyword arguments. "Stringly typed" libraries are easy to use for beginners and convenient to implement for library authors. However, they miss out on several benefits of static typing. - Push errors left: Instead of exceptions at runtime, many errors can be discovered by an IDE, type checker, or other automated tools. - Autocomplete: An IDE shows all available arguments and options and their types. This serves as quick documentation and saves typing (and typos). - Robustness: Static analysis provides an additional safety layer on top of unit tests. To achieve these benefits, it is not enough to annotate existing code with type hints. Instead, the code has to be redesigned from the ground up with static typing in mind. This talk shares insights from redesigning "optimagic" (a Python package for numerical optimization) to get the most out of type hints. While preserving backward compatibility, we establish new recommended ways of using the library. Users who adopt the new workflow benefit immediately from better autocomplete, better static analysis, and fewer runtime errors. Most importantly, we do not compromise on simplicity for beginner users. While the talk focuses on benefits for users, maintainers and library authors also benefit via improved maintainability and extensibility. The lessons learned from the redesign are not specific to the case study. They apply to any package in the scientific ecosystem that lets users select between multiple algorithms or methods, all of which have different optional arguments. Prime examples are plotting libraries, ODE solvers, Numerical Integration packages, and many more. Moreover, they are relevant for anyone adopting type hints in large existing codebases. false https://pretalx.com/euroscipy-2024/talk/CETWRS/ https://pretalx.com/euroscipy-2024/talk/CETWRS/feedback/ Room 6 Talk (25 mins + Q&A) 2024-08-28T11:05:00+02:00 11:05 00:30 Hi! Have you ever wished your pure Python libraries were faster? Or wanted to fundamentally improve a Python library by rewriting everything in a faster language like C or Rust? Well, wish no more... NetworkX's backend dispatching mechanism redirects your plain old NetworkX function calls to a FASTER implementation present in a separate backend package by leveraging the Python's [`entry_point`](https://packaging.python.org/en/latest/specifications/entry-points) specification! NetworkX is a popular, pure Python library used for graph(aka network) analysis. But when the graph size increases (like a network of everyone in the world), then NetworkX algorithms could take days to solve a simple graph analysis problem. So, to address these performance issues this backend dispatching mechanism was recently developed. In this talk, we will unveil this dispatching mechanism and its implementation details, and how we can use it just by specifying a `backend` kwarg like this: >>> nx.betweenness_centrality(G, backend=“parallel”) or by passing the backend graph object(type-based dispatching): >>> H = nxp.ParallelGraph(G) >>> nx.betweenness_centrality(H) We'll also go over the limitations of this dispatch mechanism. Then we’ll use the example of nx-parallel as a guide to building our own custom NetworkX backend. And then, using NetworkX's existing test suite, we'll test this backend that we build. Ending with a quick dive into the details of the nx-parallel backend. euroscipy-2024-50930-understanding-networkx-s-api-dispatching-with-a-parallel-backend Community, Education, and Outreach Erik WelchAditi Juneja en ## Ideal flow In the first few minutes, we will familiarize ourselves with the NetworkX library and graph analysis in general. Subsequently, we will delve into the growing demand for faster graph algorithms driven by numerous PRs proposing to include parallel implementations for various algorithms in NetworkX. Moving on to a quick demo of the performance limitations of existing NetworkX algorithms such as `betweenness_centrality` and `square_clustering` when applied to larger SNAP graph datasets, underscoring the critical necessity for more efficient implementations. Balancing this need for more efficient and faster algorithms with the core values of NetworkX-- to remain free from any external dependencies and to uphold its pure Python nature, led to the development of a new backend dispatching system in NetworkX. Next, we will understand what `entry_points` are and how NetworkX utilizes them to discover the backend packages and then redirect the NetworkX function calls to the backend implementations according to the specified backend. I will then delve into the details of the features a backend developer can utilize, like how they can use NetworkX's existing testing suite to test their backend just by setting the `NETWORKX_TEST_BACKEND` environment variable. Using the nx-parallel backend as an example, I will explain these implementation details and features. Finally, I will provide a brief demo on building your own backend. In the last few minutes, we will get to know the nx-parallel backend that runs NetworkX's single-core graph algorithms on multiple CPU cores using [joblib](https://joblib.readthedocs.io/en/stable/generated/joblib.Parallel.html). We will discuss its implementation details and observe the parallel processes running concurrently and their CPU core usage distribution in the Activity Monitor. Next, we will explore the concept of chunking, and learn how and when it helps in parallel computing, particularly in the context of graph algorithms. Many algorithms in nx-parallel are generator functions, so we’ll also go over how chunking is done for generator functions. In the end, we’ll engage in a quick demo comparing nx-parallel and NetworkX using a large graph dataset with custom chunking enabled. We will end with a summary of other NetworkX backends and some future to-dos for NetworkX’s backend dispatching and the nx-parallel backend. And then finally conclude with an interactive Q&A. Refer to the linked poster below for more! Thank you :) false Slides Poster https://pretalx.com/euroscipy-2024/talk/QLVBYY/ https://pretalx.com/euroscipy-2024/talk/QLVBYY/feedback/ Room 6 Talk (15 mins + Q&A) 2024-08-28T11:40:00+02:00 11:40 00:20 Bayesian optimization is a powerful technique for optimizing black-box, costly-to-evaluate functions, widely applicable across diverse fields. However, Gaussian process (GP) models commonly used in Bayesian optimization struggle with functions defined on categorical or mixed domains, limiting optimization in scenarios with numerous categorical inputs. In this talk, we present a solution by leveraging ensemble models for probabilistic modelling, providing a robust approach to optimize functions with categorical inputs. We showcase the effectiveness of our method through a Bayesian optimization setup implemented with the BoTorch library, utilizing probabilistic models from the XGBoostLSS framework. By integrating these tools, we achieve efficient optimization on domains with categorical variables, unlocking new possibilities for optimization in practical applications. euroscipy-2024-50803-enhancing-bayesian-optimization-with-ensemble-models-for-categorical-domains Data Science and Visualisation Ilya Komarov en Bayesian optimization (BO) is a powerful method for optimizing black-box, costly-to-evaluate functions, with applications across various fields. These include hyperparameter tuning for complex machine learning models, designing better-tasting beverages, geological carbon sequestration, and developing new chemical products. BO algorithms rely on two key components: a probabilistic model and an acquisition function. The probabilistic model predicts the target variable's distribution at each point in the predictor space. At the same time, the acquisition function scores these distributions to guide the selection of the following evaluation points, aiming for efficient optimization. The Gaussian Process (GP) is BO’s most popular probabilistic model. However, GPs have limitations. They struggle with functions defined on categorical or mixed domains, making them less effective when numerous categorical inputs are involved. Optimizing with GPs requires a careful choice of likelihood, kernel functions, and priors, posing a risk of mismodeling. Additionally, GPs can’t natively describe functions with conditional spaces, their training time increases polynomially with the number of training samples, and the popular implementation of BO with GP (BoTorch) does not support discontinuous polytope inference. An intelligent choice of probabilistic model can address these limitations. In this talk, we benchmark tree-based ensemble probabilistic models against GPs on several corner cases. We explore constrained optimization on mixed domains and functions with conditional spaces. We compare training and inference times, scaling with dimensionality, and optimization performance. We consider existing solutions, such as GPyOpt with sklearn's RandomForestRegressor, and our implementation integrating BO with the BoTorch library using probabilistic models from the XGBoostLSS framework. Through practical examples, we will demonstrate the effectiveness of tree-based probabilistic models for BO and showcase how our approach can unlock new possibilities for optimization in real-world applications. false Talk material https://pretalx.com/euroscipy-2024/talk/SPUZPK/ https://pretalx.com/euroscipy-2024/talk/SPUZPK/feedback/ Room 6 Talk (25 mins + Q&A) 2024-08-28T13:20:00+02:00 13:20 00:30 The conversation about reproducibility is usually focused on how to make research workflows (more) reproducible. Here, we consider it from the opposite perspective, and ask: How feasible is it, in practice, to reproduce research which is meant to be reproducible? Is it even done or attempted? We provide a detailed account of such an attempt, trying to reproduce some segmentation results for 3D microscopy images of a developing mouse embryo. The original research is a monumental work of bioimaging and analysis at the single-cell level, published in *Cell* in 2018, alongside with all the necessary research artifacts. Did we succeed in this attempt? As we share the joys and pains of this journey, many questions arise: How do reviewers assess the reproducibility claims exactly? Incentivizing reproducible research is still an open problem, since it is so much more costly (in time) to produce. And how can we incentivize those who test reproducibility? Not only is it costly to set up computational environments and execute data-intensive scientific workflows, but it may not appear as rewarding at first thought. In addition, there is a human factor: It is thorny to show authors that their publication does not hold up to their reproducibility claims. euroscipy-2024-50908-the-joys-and-pains-of-reproducing-research-an-experiment-in-bioimaging-data-analysis Data Science and Visualisation Marianne Corvellec en In this presentation, I would like to share my personal experience trying to reproduce a bioimaging data analysis workflow. The starting point was my fascination with the live imaging of developing mouse embryos in the accompanying videos of a 2018 research paper [1]. These incredible images show the organism's individual cells (nuclei being marked with a fluorescent protein) dividing and migrating over time, in three spatial dimensions. The authors of this research developed not only imaging but also analysis tools, in order to achieve accurate cell segmentation, cell tracking, detection of cell divisions, registration across multiple embryos, and more. As a maintainer of scikit-image, and just for starters, I was curious to try out some of our classical (as opposed to machine-learning) segmentation workflows on these 3D biomedical images. After all, the paper is open access, it comes with supplementary materials which point to data and software repositories... and its authors are kind and helpful! The latter point proved to be extremely important, especially when it came to getting and using the data. I would like to commend these researchers, who are so conscientious and generous in their sharing. In April 2023, the scikit-image team selected two Outreachy interns [2] to work on two different projects, including one on narrative documentation to expand our gallery with biomedical examples. The additional workforce was welcome, considering the challenges we encountered every step along the way. The datasets are available in the Image Data Resource (IDR) [3], which is fairly standard practice in the field, but it turns out that downloading data from the IDR is not trivial [4, 5, 6 and references therein]. For long-term cell tracking, the authors of [1] developed a framework named TGMM (Tracking with Gaussian Mixture Models). If you pass it a single frame, it computes a cell segmentation for this frame. Admittedly, the question of comparing this segmentation result with another one, say, obtained in pure Scientific Python, goes beyond that of reproducibility. We present a segmentation workflow based on SciPy and scikit-image, and compare its results with the TGMM's (although this would rather fall under the "Scientific Applications" track). The datasets are available in KLB format [7], which is not much in use anymore. Nowadays, a similar study would probably publish their data in the Zarr format, which is popular in bioimaging [8]. This would make it much easier to load the data into various analysis tools. For example, the `dask.array.from_zarr` function [9] might be convenient to work in Python. But is it fair to ask that the data be available in Zarr, just because the ecosystem has changed since 2018? Who would take care of converting (large) published datasets, assuming there is an actual demand for re-using them? To read a dataset in Python, I used the `pyklb` package [10], which provides a Python wrapper for the KLB file format but is not maintained anymore. Unsurprisingly, I had to use a custom-made environment just to be able to install `pyklb`. The documentation of all the tweaking that ensued lives in GitHub issues [11] for now. I will compile it, along with the rest of my logbook, and share it in a dedicated repository [12], so the reproducibility loop is complete. Getting the TGMM software to run on my PC proved painful but ultimately possible (which made it almost joyful)! Research-style documentation tends to be scattered and out-of-date (e.g., [13]), which is completely understandable: In research groups, people and funding cycles come and go, so who really would be able to maintain software published (shared) as part of a reproducible study? For exemple, will these tiny pull requests [14] ever be seen, let alone merged? Should there be some kind of community responsibility here? Shifting the perspective, should we consider that research software is significantly different from 'regular' software, in the sense that it goes through an indefinite code freeze from the moment it is 'released?' It seems fair (and it is sufficient for reproducibility) that, for a given study, research artifacts would be published as a snapshot only. Note that the TGMM software provides Docker images [15]. This presentation definitely brings more questions than it offers answers. We look forward to hearing what the audience could share in terms of ideas, resources, and experiences. We would love to know of other attempts at reproducing published open research. We suspect there is a special scenario in which the person reproducing the work is the future self of the original author! [1] https://doi.org/10.1016/j.cell.2018.09.031 [2] https://www.outreachy.org/ (accessed 2024-05-25) [3] "The Image Data Resource (IDR) is a public repository of image datasets from published scientific studies, where the community can submit, search and access high-quality bio-image data." https://idr.openmicroscopy.org (accessed 2024-05-25) [4] https://idr.openmicroscopy.org/about/download.html (accessed 2024-05-25) [5] https://github.com/IDR/idr.openmicroscopy.org/pull/193 (accessed 2024-05-25) [6] https://forum.image.sc/t/permission-denied-when-trying-to-download-idr-openmicroscopy-at-fasp/88907 (accessed 2024-05-25) [7] KLB 2.0 block-based lossless image compression file format https://bitbucket.org/fernandoamat/keller-lab-block-filetype (accessed 2024-05-25) [8] https://gerbi-gmb.de/2023/10/02/next-generation-file-formats-for-bioimaging/ (accessed 2024-05-25) [9] https://docs.dask.org/en/latest/generated/dask.array.from_zarr.html (accessed 2024-05-25) [10] https://github.com/bhoeckendorf/pyklb (accessed 2024-05-25) [11] https://github.com/bhoeckendorf/pyklb/issues/11 (accessed 2024-05-25) [12] https://github.com/mkcor/repro-tgmm (accessed 2024-05-25) [13] https://bitbucket.org/fernandoamat/tgmm-paper/src/master/doc/ (accessed 2024-05-25) [14] https://bitbucket.org/fernandoamat/tgmm-paper/pull-requests/ (accessed 2024-05-25) [15] https://bitbucket.org/fernandoamat/tgmm-paper/src/master/doc/new/docs/user-guide/docker.md (accessed 2024-05-25) false https://pretalx.com/euroscipy-2024/talk/BXNEY8/ https://pretalx.com/euroscipy-2024/talk/BXNEY8/feedback/ Room 6 Talk (25 mins + Q&A) 2024-08-28T13:55:00+02:00 13:55 00:30 This session introduces PyFixest, an open source Python library inspired by the "fixest" R package. PyFixest implements fast routines for the estimation of regression models with high-dimensional fixed effects, including OLS, IV, and Poisson regression. The library also provides tools for robust inference, including heteroscedasticity-robust and cluster robust standard errors, as well as the wild cluster bootstrap and randomization inference. Additionally, PyFixest implements several routines for difference-in-differences estimation with staggered treatment adoption. PyFixest aims to faithfully replicate the core design principles of "fixest", offering post-estimation inference adjustments, user-friendly syntax for multiple estimations, and efficient post-processing capabilities. By making efficient use of jit-compilation, it is also one of the fastest solutions for regressions with high-dimensional fixed effects. The presentation will argue why there is a need for another regression package in Python, cover PyFixest's functionality and design philosophy, and discuss future development prospects. euroscipy-2024-50927-mostly-harmless-fixed-effects-regression-in-python-with-pyfixest Data Science and Visualisation Alexander Fischer en This session introduces PyFixest, an open source Python library inspired by the "fixest" R package. PyFixest implements fast routines for the estimation of regression models with high-dimensional fixed effects, including OLS, IV, and Poisson regression. The library also provides tools for robust inference, including heteroscedasticity-robust and cluster robust standard errors, as well as the wild cluster bootstrap and randomization inference. Additionally, PyFixest implements several routines for difference-in-differences estimation with staggered treatment adoption. PyFixest aims to faithfully replicate the core design principles of "fixest", offering post-estimation inference adjustments, user-friendly syntax for multiple estimations, and efficient post-processing capabilities. By making efficient use of jit-compilation, it is also one of the fastest solutions for regressions with high-dimensional fixed effects. The presentation will argue why there is a need for another regression package in Python, cover PyFixest's functionality and design philosophy, and discuss future development prospects. false https://pretalx.com/euroscipy-2024/talk/UGJ3HQ/ https://pretalx.com/euroscipy-2024/talk/UGJ3HQ/feedback/ Room 6 Talk (15 mins + Q&A) 2024-08-28T14:25:00+02:00 14:25 00:20 In the ever-evolving landscape of data science, accurate uncertainty quantification is crucial for decision-making processes. Conformal Prediction (CP) stands out as a powerful framework for addressing this challenge by providing reliable uncertainty estimates alongside predictions. In this talk, I'll delve into the world of Conformal Prediction, with a focus on the MAPIE Python library, offering a comprehensive understanding of its advantages and practical applications. euroscipy-2024-50894-conformal-prediction-with-mapie-a-journey-into-reliable-uncertainty-quantification Data Science and Visualisation Claudio Giorgio Giancaterino en -Advantages and Fundamentals concepts of Conformal Prediction Uncertainty is an inherent aspect of real-world data, and accurate quantification is vital for making informed decisions. Conformal Prediction offers a principled approach to estimate the uncertainty associated with predictions, providing users with more reliable and actionable insights. -Types of conformal predictors Not all conformal predictors are created equal. I'll give an introduction of different types of CP predictors. -MAPIE python library I'll present MAPIE (Model Agnostic Prediction Interval Estimator), a Python library that simplifies the implementation of Conformal Prediction. -Practical example on tabular data To bring theory into practice, I'll walk through an use case using tabular data. false https://pretalx.com/euroscipy-2024/talk/NH7LGF/ https://pretalx.com/euroscipy-2024/talk/NH7LGF/feedback/ Room 5 Maintainer track 2024-08-28T10:30:00+02:00 10:30 00:45 OpenGL is old and on a path to being deprecated. Modern GPU API's like Vulkan and Metal solve most problems that plague OpenGL, and higher abstractions like wgpu / WebGPU provide a modern interface to control GPU hardware. The way that these work is much more pleasant to work with, and also provides performance benefits, especially for Python. euroscipy-2024-50962-opengl-is-dying-let-s-talk-about-webgpu Data Science and Visualisation Almar Klein en I'll talk about the problems that plague OpenGL, and how modern API's like Vulkan, Metal, and DirectX12 solve these. Since these API's are still very low level, we need abstractions to provide a friendlier API. The most promising abstraction is WebGPU, which also comes with a corresponding C-API (webgpu.h). The wgpu-py library provides a Python wrapper. This talk is for anyone who wants to learn more about the current state of GPU API's and/or is interested in wgpu / WebGPU. I'll briefly discuss the GPU programming model, but a basic understanding of GPU's is assumed. Apart from obvious applications in graphics and visualization, there are also opportunities to use wgpu for compute tasks, i.e. as an alternative to Cuda. I'll keep the talk relatively short, so we have plenty of time for questions and discussions. false https://pretalx.com/euroscipy-2024/talk/QMV8P3/ https://pretalx.com/euroscipy-2024/talk/QMV8P3/feedback/ Room 5 Maintainer track 2024-08-28T11:15:00+02:00 11:15 00:45 Learn more about the Scientific Python project (https://scientific.python.org): what it aims to achieve (helping the developer community), recent progress that has been made, and how to become involved. euroscipy-2024-54012-scientific-python Community, Education, and Outreach Jarrod MillmanStéfan van der Walt en The scientific Python ecosystem comprises foundational libraries like NumPy and SciPy, technique-specific libraries like scikit-learn, NetworkX, and scikit-image, and domain-specific libraries such as PyHEP and AstroPy. The Scientific Python project is an effort to better coordinate and support the community of scientific Python ecosystem developers. In this interactive talk, we give project updates, and invite the community to become more involved in our joint efforts to improve the ecosystem for its developers. false https://pretalx.com/euroscipy-2024/talk/SYXTMZ/ https://pretalx.com/euroscipy-2024/talk/SYXTMZ/feedback/ Room 5 Maintainer track 2024-08-28T13:55:00+02:00 13:55 00:50 Goals: - Share tips, tricks and best practices for configuring the build backend of a Python package with compiled (Cython/C/C++/Rust/Fortran) code - Identify shared needs between packages, and discuss gaps in current build backends, documentation, or shared infrastructure Topics: - Goals to aim for in your build config (and how to achieve them): - Faster builds and relevant tooling like profiling, - Build logs that actually help when diagnosing issues, - How to debug build failures effectively, - How to check for and visualize build dependencies, - Ensuring builds are reproducible, - Approaches to reducing binary size, - CI config ideas to guard against regressions - Recent build-related developments & a post-distutils world - What are the most pressing pain points for maintainers? euroscipy-2024-54014-change-of-program-informal-discussions-about-switching-build-backends Scientific Applications Ralf Gommers en Goals: - Share tips, tricks and best practices for configuring the build backend of a Python package with compiled (Cython/C/C++/Rust/Fortran) code - Identify shared needs between packages, and discuss gaps in current build backends, documentation, or shared infrastructure Topics: - Goals to aim for in your build config (and how to achieve them): - Faster builds and relevant tooling like profiling, - Build logs that actually help when diagnosing issues, - How to debug build failures effectively, - How to check for and visualize build dependencies, - Ensuring builds are reproducible, - Approaches to reducing binary size, - CI config ideas to guard against regressions - Recent build-related developments & a post-distutils world - What are the most pressing pain points for maintainers? false https://pretalx.com/euroscipy-2024/talk/LMSJ8Z/ https://pretalx.com/euroscipy-2024/talk/LMSJ8Z/feedback/ Room 7 Keynote 2024-08-29T09:00:00+02:00 09:00 01:00 Open source software is here for everyone - but how are we making sure that everyone has equal access? In this keynote I will discuss how to lower barriers of entry for new contributors - and the many facets to this: documentation, community, guidelines, and tools. I will share my personal motivations for contributing to open-source software and my journey over the past five years and all of its learnings. euroscipy-2024-54210-just-contribute Community, Education, and Outreach Wolf Vollprecht en Open source software is here for everyone - but how are we making sure that everyone has equal access? In this keynote I will discuss how to lower barriers of entry for new contributors - and the many facets to this: documentation, community, guidelines, and tools. I will share my personal motivations for contributing to open-source software and my journey over the past five years and all of its learnings. false https://pretalx.com/euroscipy-2024/talk/9EUT78/ https://pretalx.com/euroscipy-2024/talk/9EUT78/feedback/ Room 7 Talk (25 mins + Q&A) 2024-08-29T10:30:00+02:00 10:30 00:30 Python has many high quality optimization algorithms but they are scattered across many different packages. Switching between packages is cumbersome and time consuming. Other languages are ahead of Python in this respect. For example, `Optimization.jl` provides a unified interface to more than 100 optimization algorithms and is widely accepted as a standard interface for optimization in Julia. In this talk, we take stock of the existing optimization ecosystem in Python and analyze pain points and reasons why no single package has emerged as a standard so far. We use these findings to derive desirable features a Python optimization package would need to unify the ecosystem. We then present optimagic, a NumFocus affiliated Project with the goal of unifying the Python optimization ecosystem. Optimagic provides a common interface to optimization algorithms from scipy, NlOpt, pygmo, and many other libraries. The minimize function feels familiar to users of scipy.optimize who are looking for a more extensive set of supported optimizers. Advanced users can use optional arguments to configure every aspect of the optimization, create a persistent log file, turn local optimizers global with a multistart framework, and more. Finally, we discuss an ambitious roadmap for improvements, new features, and planned community activities for optimagic. euroscipy-2024-50868-optimagic-can-we-unify-python-s-numerical-optimization-ecosystem Community, Education, and Outreach Janos Gabler en Numerical optimization is a large field with applications in engineering, statistics, data science, and many other disciplines. The fundamental goal is always the same: Find a set of parameters that makes a number large or small (potentially fulfilling some constraints). Unfortunately, no single algorithm exists that can solve all optimization problems. Therefore, doing optimization in practice usually involves a lot of trial and error until one finds an optimizer that works well for specific problem characteristics. The good news is that many high-quality optimization algorithms are implemented in Python or have Python bindings. The bad news is that they are scattered across many different packages. Switching between packages is expensive, as each package has its own way of specifying problems and calling optimizers. Other languages are ahead of Python in this respect. For example, `Optimization.jl` provides a unified interface to more than 100 optimization algorithms and is widely accepted as a standard interface for optimization in Julia. In this talk, we take stock of the existing optimization ecosystem in Python and analyze pain points and reasons why no single package has emerged as a standard so far. We use these findings to derive desirable features a Python optimization package would need to unify the ecosystem. We then present optimagic, a NumFocus affiliated Project with the goal of unifying the Python optimization ecosystem. Optimagic provides a common interface to optimization algorithms from scipy, NlOpt, pygmo, and many other libraries. The minimize function feels familiar to users of scipy.optimize who are looking for a more extensive set of supported optimizers. Advanced users can use optional arguments to configure every aspect of the optimization, create a persistent log file, turn local optimizers global with a multistart framework, and more. Finally, we discuss an ambitious roadmap for improvements, new features, and planned community activities for optimagic. false https://pretalx.com/euroscipy-2024/talk/NGECXK/ https://pretalx.com/euroscipy-2024/talk/NGECXK/feedback/ Room 7 Talk (25 mins + Q&A) 2024-08-29T11:00:00+02:00 11:00 00:30 Foundation models are here for forecasting! This will conclusively solve all forecasting problems with a one-model-fits-all approach! Or … maybe not? Fact is, an increasingly growing number of foundation models for time series and forecasting hitting the market. To innocent end users, this situation raises various challenges and questions. How do I integrate the models as candidates into existing forecasting workflows? Are the models performant? How do they compare to more classical choices? Which one to pick? How to know whether to “upgrade”? At sktime, we have tried so you don’t have to! Although you will probably be forced to anyway, but even then, it’s worth sharing experiences. Our key challenges and findings are presented in this talk – for instance, the unexpected fragmentation of the ecosystem, difficulties in evaluating the models fairly, and more. (sktime is an openly governed community with neutral point of view. You may be surprised to hear that this talk will not try to sell you a foundation model) euroscipy-2024-50932-forecasting-foundation-models-evaluation-and-integration-with-sktime-challenges-and-outcomes Machine and Deep Learning Franz KiralyBenedikt Heidrich en During the talk, we will expand on a number of challenges we believe end users will typically face, and how we approached them: - API fragmentation of different foundation models. Different models use vastly different interfaces - even if weights are available on model sharing platforms, they do not come with consistent specifications. This is a substantial, underestimated challenge, and providing consistent APIs is difficult (but sktime helps!) - Customer lock-in dynamics. Many providers would like to tie you to their model and upsell. This disincentivizes interoperability and comparability – between competitor solutions, but also with classical models, baselines, or second-to-latest generation models. But comparability and interoperability are in the interest of the end user - and can be addressed with prudent architectural decisions. - Handling of nested software backend layers. Using a foundation model in a consistent API requires one to juggle multiple layers – model framework, deep learning backend, data backend, model marketplace, fine-tuning functionality. It is even more difficult to design a coherent software API for your production system. Learnings from sktime integrations are presented. Finally, we will report results of tests and evaluations, including aspects of software integration, performance, and trustworthiness – exclusive at EuroSciPy 2024. false https://pretalx.com/euroscipy-2024/talk/XBXX89/ https://pretalx.com/euroscipy-2024/talk/XBXX89/feedback/ Room 7 Talk (15 mins + Q&A) 2024-08-29T11:30:00+02:00 11:30 00:20 The Mission Support System (MSS) is an open source software package that has been used for planning flight tracks of scientific aircraft in multiple measurement campaigns during the last decade. It consists of many components, a data-retrieval tool chain, a wms server which creates 2-D figures from 4-D meterogical data. A client application for displaying the figures in combination with the planned flight track and other data. For data exchange between participants a collaboration server is used. The talk describes how we used these components for a campaign. euroscipy-2024-51597-the-mission-support-system-and-its-use-in-planning-an-aircraft-campaign Scientific Applications Reimar Bauer en The presentation shows why we need software and data to cooperatively manage an aircraft for our missions. We have to figure for a group of instruments the optimal flight path through the upper trosposphere / lower stratosphere. After a short introduction into the complexity of aircraft campaigns I want to show the components of the software package. - our documentation - a data retrieval tool chain for 4-D model forecast data. - Based on the data and the aim a modified WMS Server which shows a Side View and a Linear View upon the common Top View - An UI which interacts with this kind of server and the possibility to work collaborativly on a flight track. - As summary how the software was used in a recent campaign false https://pretalx.com/euroscipy-2024/talk/GQN8AF/ https://pretalx.com/euroscipy-2024/talk/GQN8AF/feedback/ Room 7 Talk (25 mins + Q&A) 2024-08-29T13:20:00+02:00 13:20 00:30 This talk introduces a new render engine for Python, called pygfx (pronounced "py-graphics"). Its purpose is to bring powerful and reliable visualization to the Python world. Since pygfx is built on wgpu, it has superior performance and reliability compared to OpenGL-based solutions. It is also designed to be versatile: with its modular architecture, one can assemble graphical scenes for diverse applications, ranging from scientific visualization to video games. euroscipy-2024-50742-wgpu-and-pygfx-next-generation-graphics-for-python Data Science and Visualisation Almar Klein en ### Intro This talk introduces a new render engine for Python, called pygfx (pronounced "py-graphics"), as well as our Python wrapper for wgpu. ### Purpose The purpose of pygfx is to bring powerful and reliable visualization to the Python world. Since pygfx is built on wgpu, it has superior performance and reliability compared to OpenGL-based solutions. It is also designed to be versatile: with its modular architecture, one can assemble graphical scenes for diverse applications, ranging from scientific visualization to video games. ### What you can expect In this talk we will touch on a few technical details related to GPU programming, but we will explain these in the context of how these affect the graphics that we need in the scientific Python community. We'll discuss how wgpu has enabled us to introduce features in pygfx that we could only dream of previously. And we will show how wgpu and pygfx fit into the scientific ecosystem. false https://pretalx.com/euroscipy-2024/talk/GKYTSY/ https://pretalx.com/euroscipy-2024/talk/GKYTSY/feedback/ Room 7 Talk (25 mins + Q&A) 2024-08-29T13:55:00+02:00 13:55 00:30 Fast interactive visualization remains a considerable barrier in analyses pipelines for large neuronal datasets. Here, we present fastplotlib, a scientific plotting library featuring an expressive API for very fast visualization of scientific data. Fastplotlib is built upon pygfx which utilizes the GPU via WGPU, allowing it to interface with modern graphics APIs such as Vulkan for fast rendering of objects. Fastplotlib is non-blocking, allowing for interactivity with data after plot generation. Ultimately, fastplotlib is a general purpose scientific plotting library that is useful for the fast and live visualization and analysis of complex datasets. euroscipy-2024-50890-fastplotlib-a-high-level-library-for-ultra-fast-visualization-of-large-datasets-using-modern-graphics-apis Data Science and Visualisation Kushal KolarCaitlin Lewis en Over the past decade, advanced analyses pipelines have been developed for the analysis of large datasets. However, fast visualization and live interactivity during data collection remains challenging. While current tools within the Python plotting ecosystem allow for interactive data visualization, they either fail to leverage modern GPUs efficiently, lack intuitive APIs for rapid prototyping, or require users to write their own shaders. Additionally, other popular plotting libraries, such as bokeh and matplotlib, are not geared towards fast interactive visualization with millions of objects. Given these challenges with current visualization tools, the need for a modern GPU-driven interactive plotting library exists. In this presentation, we will go through the technical details, as well as a brief demo on how fastplotlib makes fast interactive visualization of complex datasets possible. We will demonstrate the broad applicability of fastplotlib as a fast, general-purpose plotting library. Fastplotlib is built on top of pygfx which is a cutting edge Python rendering engine that utilizes WGPU, which can efficiently leverage modern GPU and CPU hardware. WGPU is the successor to OpenGL and features a low overhead with respect to the amount of code per-draw-per-object allowing for speed even when rendering millions of objects. Pygfx is also non-blocking, which allows for interactivity and modification of already drawn objects. Fastplotlib utilizes the pygfx rendering library for fast visualization with an expressive API for scientific visualization. The benefits of fastplotlib are that it reduces boilerplate code which allows users to focus on their data without having to manage the underlying rendering process. Additionally, fastplotlib allows for animations as well as high-level interactivity among plots, which can be combined with lazy loading and lazy compute of very large datasets that are hundreds of gigabytes or terabytes in size. Furthermore, fastplotlib can be used in jupyter notebooks, allowing it to be used on cloud computing and other remote infrastructures for streaming visualizations of extremely large datasets. In total, these unique features and the underlying architecture create a plotting library that is fast, easy to use, and multifaceted. false https://pretalx.com/euroscipy-2024/talk/3RENPJ/ https://pretalx.com/euroscipy-2024/talk/3RENPJ/feedback/ Room 7 Talk (25 mins + Q&A) 2024-08-29T14:30:00+02:00 14:30 00:30 Napari is an interactive n-dimensional image viewer for Python. It is able to rapidly render and interactively visualize almost any array like image data. Additionally, napari can overlay derived data, such as segmentations, points, polygons, surfaces and more. Each of these data exists as a layer in the napari viewer, which allows fine control over how the data is displayed. Furthermore, derived data can be edited. Together with the capability of writing plugins, napari lets you seamlessly weave exploration, computation, and annotation in common and custom image analysis workflows. euroscipy-2024-50907-napari-multi-dimensional-image-visualization-annotation-and-analysis-in-python Data Science and Visualisation Grzegorz BokotaWouter-Michiel Vierdag en Napari is an interactive n-dimensional image viewer for Python. In contrast to matplotlib it can quickly visualize large data even when the data has more than 2 dimensions. Data can also be larger than RAM as napari can natively perform chunked loading when the array is chunked. The napari canvas can be 2D or 3D. When you give napari an array with more dimensions than the canvas, it will automatically create sliders for those additional dimensions, allowing you to rapidly explore your data to the full extent, rather than a few sampled slices. Image analysis and visualization involves more than images though: feature detection algorithms result in points, segmentation results in label images, annotation results in shapes such as rectangles or polygons, and more. Napari provides layers that can be displayed on top of each other or side by side. Layers can also be controlled programmatically, which allows users to create workflows compatible with other scientific Python libraries to gain a rapid understanding of the performance of their algorithms, identifying strengths and pinpointing areas for improvement. Sometimes, image analysis algorithms get you this far, but not quite far enough. In such cases, it’s useful to manually curate their output, and then continue with downstream steps of an analysis. Napari provides editing tools for its layer types, allowing one for example to add missing points to the output of a peak detection algorithm, remove incorrect ones, paint over incorrect parts of a segmentation, or draw polygons around missed objects of interest. The resulting data points are saved in standard Scientific Python data structures, such as NumPy or Zarr arrays. This design makes it easy to seamlessly weave together image exploration, image computation, processing, and analysis, and data annotation, curation, and editing. Napari also provides a plugin interface, allowing developers to extend napari’s capabilities, providing users with novel ways to interact with their data. Because napari provides both a library accessible within Python, IPython, and Jupyter, and a standalone executable script, we have even found that napari plugins can be an effective way to help collaborators run Python image analysis workflows without having to launch Python. Because most new algorithms are being developed in Python, it is straightforward to try out new algorithms and assess how they perform using napari or even write a plugin that allows others to easily try as well, even in case of the user not having coding experience. Napari is still being improved. We are actively working on async slicing and rendering, multicanvas views, layer groups etc.. Developments are guided by the user community. We actively encourage contributions by organizing weekly community meetings, napari code cafés and paired coding sessions. In this talk we will cover the key features of napari, including its layer-based approach to handling image data, interactive annotation and segmentation tools, and show real-time performance capabilities. Additionally, we will demonstrate how napari can be leveraged in various scientific workflows, highlighting use cases in biology, neuroscience, and medical imaging. Attendees will gain insights into how napari can enhance their data visualization tasks and streamline analysis pipelines. false https://pretalx.com/euroscipy-2024/talk/3HCQFS/ https://pretalx.com/euroscipy-2024/talk/3HCQFS/feedback/ Room 7 Talk (25 mins + Q&A) 2024-08-29T15:30:00+02:00 15:30 00:30 CPython 3.13 will be released in October 2024 and has been in beta since May 2024. One of its most awaited features is the possibility to remove the GIL (Global Interpreter Lock) through a compile-time flag. In this talk we will explain the relevance of free-threaded CPython for the Scientific Python ecosystem, what already works, some of the caveats, and how to try it out on your favourite use case. In particular we will discuss: - the historic effort in the scikit-learn project to add Continuous Integration for the `nogil` fork of CPython 3.9, and the kind of issues that were surfaced - the ongoing effort in the Scientific Python ecosystem (Numpy, Scipy, scikit-learn, etc ...) to test free-threaded CPython 3.13 and fix issues along the way - how a typical scikit-learn grid-search use case can benefit from free-threaded CPython - how to try out free-threaded CPython on your favourite use case - possible future developments euroscipy-2024-51413-free-threaded-aka-nogil-cpython-in-the-scientific-python-ecosystem-status-and-road-ahead Machine and Deep Learning Loïc Estève en CPython 3.13 will be released in October 2024 and has been in beta since May 2024. One of its most awaited features is the possibility to remove the GIL (Global Interpreter Lock) through a compile-time flag. In this talk we will explain the relevance of free-threaded CPython for the Scientific Python ecosystem, what already works, some of the caveats, and how to try it out on your favourite use case. In particular we will discuss: - the historic effort in the scikit-learn project to add Continuous Integration for the `nogil` fork of CPython 3.9, and the kind of issues that were surfaced - the ongoing effort in the Scientific Python ecosystem (Numpy, Scipy, scikit-learn, etc ...) to test free-threaded CPython 3.13 and fix issues along the way - how a typical scikit-learn grid-search use case can benefit from free-threaded CPython - how to try out free-threaded CPython on your favourite use case - possible future developments false https://pretalx.com/euroscipy-2024/talk/DLYRXH/ https://pretalx.com/euroscipy-2024/talk/DLYRXH/feedback/ Room 7 Talk (25 mins + Q&A) 2024-08-29T16:00:00+02:00 16:00 00:30 The world of open source computer vision has never been so exciting - and so challenging. With so many options available to you, what's the best way to solve your real world problem? The questions are always the same: Do I have enough data? Which model should I choose? How can I fine-tune and optimize the hyperparameters? In collaboration with the German Aerospace Center, we investigated these questions to develop a model for quality assurance of CFRP tape laying, with only a small real data set fresh from production. We are very pleased to present a machine learning setup that can empirically answer these questions. Not only for us, but also for you - our setup can easily be transferred to your application! Dive with us into the world of Open Source machine learning tools that are perfectly tailored for your next project. Discover the seamless integration of Hugging Face Model Hub, DvC and Ray Tune. You'll also gain unique insights into the fascinating world of CFRP tape laying, specifically how well different architectures of open source models perform on our small dataset. If you want to level up your MLOps game and gain practical knowledge of the latest computer vision models and practices, this talk is a must for you. Don't miss the opportunity, and look forward to your next computer vision projects! euroscipy-2024-50494-a-comparative-study-of-open-source-computer-vision-models-for-application-on-small-data-the-case-of-cfrp-tape-laying Machine and Deep Learning Thomas FraunholzTim Köhler en The world of open-source computer vision has never been so exciting—and so challenging. With so many options available, what's the best way to solve your real-world problem? The questions are always the same: Do I have enough data? Which model should I choose? How can I fine-tune and optimize the hyperparameters? In collaboration with the German Aerospace Center, we investigated these questions to develop a model for quality assurance of CFRP tape laying, using only a small real dataset fresh from production. We are very pleased to present a machine learning setup that can empirically answer these questions. Not only for us, but also for you—our setup can easily be transferred to your application! This talk provides you with a blueprint for your own projects, focusing on a setup that allows you to improve your models in a controlled manner and compare results effectively: - We begin by examining the problem through our specific use case of CFRP tape laying and breaking it down into generic solution steps. - These solution steps are translated into a machine learning pipeline using DVC (Data Version Control). This approach saves computation time on steps where neither the data nor the source code has changed and helps to keep track of your progress and performance over time using Git. - We will explore various current model architectures available in the Hugging Face Model Hub and demonstrate how you can fine-tune them on your data using Python packages such as transformers and ray. On the topic of hyperparameter search, we will discuss the available algorithms and the most promising parameters. - Finally, we will review our results, specifically how well different architectures of open-source models perform on our small dataset. We will explore the question of how different model architectures compare and whether the largest model always gives the best results. If you want to level up your MLOps game and gain practical knowledge of the latest computer vision models and practices, this talk is a must for you. Don't miss the opportunity, and look forward to your next computer vision projects! false https://pretalx.com/euroscipy-2024/talk/JWAMDE/ https://pretalx.com/euroscipy-2024/talk/JWAMDE/feedback/ Room 6 Talk (25 mins + Q&A) 2024-08-29T10:30:00+02:00 10:30 00:30 Parallel computing is essential for many performance-critical applications. Python provides many solutions for this problem. New versions of Python will support sub-interpreters and a, currently experimental, free-threading version without the Global Interpreter Lock (GIL). This talk starts with a short overview over this topic, clarifying terms such parallel, concurrent, and distribute computing as well as CPU-bound, memory-bound, and IO-bound problems. The presentation explains how Python and its standard library support parallel programming tasks. In addition, many Python libraries provide very useful approaches and tools for parallel computing. An overview of important libraries provides guidance which library can be used for what type of parallel problem. How do Python's new features such as sub-interpreters and free-threading without the Global Interpreter Lock (GIL) impact parallel Programming in Python? This talk address this question by providing examples where these features might help to make programs simpler and/or faster. euroscipy-2024-50912-the-parallel-universe-in-python-a-time-travel-to-python-3-13-and-beyond High Performance Computing Mike Müller en Parallel computing is essential for many performance-critical applications. Python provides many solutions for this problem. New versions of Python will support sub-interpreters and a, currently experimental, free-threading version without the Global Interpreter Lock (GIL). This talk starts with a short overview over this topic, clarifying terms such parallel, concurrent, and distribute computing as well as CPU-bound, memory-bound, and IO-bound problems. The presentation explains how Python and its standard library support parallel programming tasks. In addition, many Python libraries provide very useful approaches and tools for parallel computing. An overview of important libraries provides guidance which library can be used for what type of parallel problem. How do Python's new features such as sub-interpreters and free-threading without the Global Interpreter Lock (GIL) impact parallel Programming in Python? This talk address this question by providing examples where these features might help to make programs simpler and/or faster. false https://pretalx.com/euroscipy-2024/talk/E8HD9K/ https://pretalx.com/euroscipy-2024/talk/E8HD9K/feedback/ Room 6 Talk (25 mins + Q&A) 2024-08-29T11:00:00+02:00 11:00 00:30 Python is one of the most used languages today, known for its simplicity and versatile ecosystem. For performance applications such as High Performance Computing (HPC) or any other kind of numerical computing the standard CPython implementation is often not fast enough. To address these issues, enter the fascinating world of LPython, a Python compiler designed to give you the best possible performance for numerical, array-oriented code, and can also generate code using multiple backends like LLVM, C, C++, WASM. euroscipy-2024-50824-lpython-novel-fast-retargetable-python-compiler High Performance Computing Naman Gera en In this talk, we will delve into LPython, an open-source, LLVM-based Python compiler that transposes type-annotated Python code into optimized machine code. It offers rapid Ahead-Of-Time (AOT) compilation to binaries, with an option for Just-In-Time (JIT) compilation and smooth interoperability with CPython. We'll also provide benchmarks, including a comparison with Numba for JIT compilation and Clang++ and g++ for AOT compilation. We will examine the unique qualities LPython has to offer and learn the "why" behind the compiler. Its main focus is speed, and our benchmarks will validate its competitiveness with the current cutting-edge technology. Here's the talk outline: * **Introduction** (2 mins): A warm welcome and a brief presenter introduction. An overview of the talk and learning expectations for attendees. * **What is LPython?** (2 mins): Explaining the need for yet another Python compiler. How it operates at each stage and introduction to the LCompilers family. * **Phases of compilation, Intermediate representation (ASR), Optimizations** (5 mins): A look at the internals, from Python code to parsers, Abstract Syntax Tree (AST), Abstract Semantic Representation (ASR), and backends (LLVM, C, C++, WASM, etc). This section also gives a short overview of how low- and high-level machine-independent optimizations are performed. * **Just-In-Time and Ahead-Of-Time compilation** (2 mins): Distinguishing between these two types of compilation in LPython; JIT compiles code during runtime while AOT produces binary output. * **Interoperability with CPython** (2 mins): Discuss how LPython enables seamless integration with CPython libraries. Demonstrate the `@pythoncall` and `@lpython` decorators, and their practical uses (e.g., Matplotlib for graphs). * **Online Demo** (2 mins): Live demo of code compilation using LPython in a browser. Displaying AST, ASR, C, and WAT (WebAssembly Text Format) output tabs on the website. * **Speed and Performance benchmarks** (4 mins): Presenting benchmarks against Numba, Clang++, g++, and Python itself, demonstrating LPython’s ability to run on Linux, Mac, Windows, and WebAssembly. * **Accelerated code sample** (1 min): Showcasing how to speed up a simple Python method developed for tomography. * **User-friendly developer experience** (2 mins): Displaying compiler diagnostic messages (errors, warnings, etc) and highlighting the user-friendly developer experience. * **Conclusion and takeaways** (2 mins): Recapping the main points covered in the talk. Encouraging attendees to incorporate LPython into their projects and providing practical advice on integration. Upcoming features will also be discussed. * **Q&A session** (5 mins): Responding to questions and engaging in dialogue. **Prerequisites**: Intermediate experience in Python would be helpful. An interest in learning the basics of Python compilers. **Content URLs**: LPython: https://lpython.org/ LPython blog: https://lpython.org/blog/2023/07/lpython-novel-fast-retargetable-python-compiler/ LPython GitHub: https://github.com/lcompilers/lpython/ Slides/GitHub repo: to be added later. **Speaker Info**: Naman Gera currently works as a Research Software Engineer at the UK’s national synchrotron science facility, [Diamond Light Source](https://www.diamond.ac.uk/Home/About.html), based in Harwell Science and Innovation Campus, UK. He is presenting this work done with his past employer, GSI Technology, being a part of the compiler team led by [Dr. Ondrej Certik](https://ondrejcertik.com/). When AFK, Naman likes being in nature, climbing mountains, and solo travelling to many countries. **Speaker URLs**: GitHub: https://github.com/namannimmo10 LinkedIn: https://www.linkedin.com/in/namannimmo/ Email: namangera15@gmail.com false https://pretalx.com/euroscipy-2024/talk/3K8ZXN/ https://pretalx.com/euroscipy-2024/talk/3K8ZXN/feedback/ Room 6 Talk (15 mins + Q&A) 2024-08-29T11:30:00+02:00 11:30 00:20 The array API standard is unifying the ecosystem of Python array computing, facilitating greater interoperability between array libraries, including NumPy, CuPy, PyTorch, JAX, and Dask. Find out how we are using it in SciPy to bring support for hardware-accelerated (e.g. GPU) and distributed arrays to our users, and how you can do the same in your library. euroscipy-2024-50301-the-array-api-standard-in-scipy High Performance Computing Lucas Colley en SciPy has had "support for distributed and GPU arrays" on its roadmap for over five years now. For a library built around NumPy arrays, that is easier said than done. There are other array libraries, such as CuPy, PyTorch, JAX, and Dask, which can help address these user wishes. Supporting multiple array libraries is not simple, since their APIs differ. Lengthy if-else statements in every function won't cut it - what we want is to be able to write 'array-agnostic' code which will work with multiple array libraries, without having to special-case on the input array type. The Python array API standard aims to standardise functionality that exists in most array libraries. It specifies an API which 'array-consumer' libraries can use to write array-agnostic code. In this talk, I give a brief introduction to the standard, before explaining how we are implementing support for it in SciPy, and the progress which we have made so far. Rough talk outline: - 5 mins - what is the array API standard, and why should you care? - 7 mins - what work is needed for a consumer library to adopt the array API standard, and what does that look like in SciPy? What tools are available to help? - 3 mins - current progress in SciPy & looking to the future. false Slides https://pretalx.com/euroscipy-2024/talk/BYTCSC/ https://pretalx.com/euroscipy-2024/talk/BYTCSC/feedback/ Room 6 Talk (25 mins + Q&A) 2024-08-29T13:20:00+02:00 13:20 00:30 Dask is a popular Python framework for scaling your workloads, whether you want to leverage all of the cores on your laptop and stream large datasets through memory, or scale your workload out to thousands of cores on large compute clusters. Dask allows you to distribute code using familiar APIs such as pandas, NumPy and scikit-learn or write your own distributed code with powerful parallel task-based programming primitives. In this session we will dive into the many ways to deploy Dask workloads on HPC, and how to choose the right method for your workload. Then we will dig into the accelerated side of Dask and how you can leverage GPUs with RAPIDS and Dask CUDA and use UCX to take advantage of accelerated networking like Infiniband and NVLink. euroscipy-2024-50828-accelerating-python-on-hpc-with-dask High Performance Computing Jacob Tomlinson en Dask is a popular Python framework for scaling your workloads, whether you want to leverage all of the cores on your laptop and stream large datasets through memory, or scale your workload out to thousands of cores on large compute clusters. Dask allows you to distribute code using familiar APIs such as pandas, NumPy and scikit-learn or write your own distributed code with powerful parallel task-based programming primitives. We will start by exploring the concept of adaptive clusters, which allow for dynamic scaling of resources based on the workload's demands. Adaptive clusters automatically submit and manage many jobs to an HPC queue, ensuring efficient resource utilisation and cost-effectiveness. This method is particularly useful for workloads with varying computational requirements, as it adjusts the number of active workers in real-time. Next, we will dive into using runners that leverage parallel execution environments such as MPI or job schedulers like SLURM to bootstrap Dask clusters within a single large job allocation. Submitting a single job offers some benefits (aside from the fact that HPC administrators often prefer this approach), including better node locality, as the scheduler places processes on nodes that are physically closer together. This results in more efficient communication and reduced latency. Additionally, launching all workers simultaneously ensures balanced data distribution across the cluster. The session will then shift focus to the accelerated side of Dask, demonstrating how to harness the power of GPUs to significantly boost computation speed. We will introduce Dask CUDA, part of RAPIDS, a suite of open-source libraries designed to execute end-to-end data science and analytics pipelines entirely on GPUs. By integrating Dask CUDA, users can achieve unprecedented levels of performance, particularly for data-intensive tasks such as machine learning and data preprocessing. We will also explore the advantages of using UCX (Unified Communication X) to enhance Dask's performance on HPC systems with advanced networking technologies. UCX provides a high-performance communication layer that supports various network transports, including Infiniband and NVLink. By leveraging these accelerated networking options, users can achieve lower latency and higher bandwidth, resulting in faster data transfers between Dask workers and more efficient parallel computations. Outline: - Overview of Dask - Scaling out Pandas and NumPy - Custom parallel code - Workflow engines - Machine learning and AI applications - Deploying Dask on HPC - Adaptive clusters - Fixed size runners - Accelerating Dask on HPC - RAPIDS and Dask CUDA - UCX false https://pretalx.com/euroscipy-2024/talk/89BJ9Q/ https://pretalx.com/euroscipy-2024/talk/89BJ9Q/feedback/ Room 6 Talk (25 mins + Q&A) 2024-08-29T13:55:00+02:00 13:55 00:30 In this talk we will present applied research and working code to regularize Python programs using a Structured Control Flow Graph (SCFG). This is a novel approach to rewriting programs at the source level such that the resulting (regularized) program is potentially more amenable to compiler optimizations, for example when using Numba[1] to compile Python. The SCFG representation of a program is simpler to analyze and thus significantly easier to optimize because the higher order semantic information regarding the program structure is explicitly included. This can be of great benefit to many scientific applications such as High Performance Computing (HPC), a discipline that relies heavily on compiler optimizations to turn user source code into highly performant executables. Additionally the SCFG format is a first step to representing Python programs as Regionalized Value State Dependence Graphs (RVSDGs). This is another recently proposed program representation which is expected to unlock even more advanced compiler optimizations at the Intermediary Representation (IR) level. The talk will cover an introduction to the theory of SCFGs and RVSDG and demonstrate how programs are transformed. We will start with simple Python programs containing control-flow constructs and then show both the SCFG representation and the resulting regularized result to illustrate the transformations. euroscipy-2024-50899-regularizing-python-using-structured-control-flow High Performance Computing Emergency Self-Constuct en The Control Flow Graph (CFG) is a well known and established concept in computer science and used as part of the compilation or interpretation step of all modern programming languages. Unfortunately, CFGs are not always the ideal representation for compilers to work with because they often result in arbitrarily structured graphs (“spaghetti”). This in turn can lead to compiler optimization steps not being leveraged and as a result a compiler may be unable to generate an optimal low-level representation of the program. A recent enhancement is the concept of the Structured Control Flow Graph (SCFG). In this extension to CFGs all blocks are part of special regions within the SCFG. The three possible region shapes are: linear, branch and loop. A linear region is simply a linear sequence of instructions. A Branch region is a shape where the control flow is split symmetrically and joined again. Finally, a loop region is a subgraph with a single backedge from the region's exiting latch. These shapes effectively describe all possible control flow patterns of a computer program and resulting structure offers significantly more chances for a compiler to apply transformations and optimizations, which in turn may lead to significant performance improvements. The Python package being presented is capable of constructing a CFG from Python source code input in the form of an Abstract Syntax Tree (AST). Furthermore the package can then apply two algorithmic steps known as Loop Restructuring (LR) and Branch Restructuring (BR) (as described in [2]) which convert the constructed CFG into an SCFG. Lastly, the package is able to synthesize a regularized Python program from the SCFG representation which is behaviourally equivalent to the original but is potentially easier for a compiler to work with. Essentially the package implements a program transformation at the source code level, where both input and output are runnable Python programs. Going beyond SCFGs, a novel Intermediate Representation (IR), the Regionalized Value State Dependence Graph (RVSDG), has been proposed [3]. Compared to CFGs and SCFGs, which are control-flow centric IRs, RVSDGs are data-flow centric IRs. This means that a number of common compiler transforms can be performed when the program has been converted to the RVSDG representation without having to reconstruct invariant properties post transformation. Also, this representation unlocks a number of compiler transforms, as data-flow through the program is explicitly available. Transforms will be algorithmically simpler, more elegant and computationally less expensive. Importantly, the construction of the SCFG representation for the input program is a necessary first step to constructing the full RVSDG and has significant merit in its own right. Sources on GitHub as: https://github.com/numba/numba-rvsdg Package on PyPi as: https://pypi.org/project/numba-rvsdg/ References: [1] Siu Kwan Lam, Antoine Pitrou and Stanly Seibert. Numba: A LLVM-based Python JIT Compiler. Proc. Second Workshop on the LLVM Compiler Infrastructure in HPC, pp. 1-6, ACM. 2015 [2] Helge Bahmann, Nico Reissmann, Magnus Jahre, and Jan Christian Meyer. Perfect reconstructability of control flow from demand dependence graphs. ACM Transactions on Architecture and Code Optimization, 11(4):66:1–66:25, 2015. [3] Nico Reissmann, Jan Christian Meyer, Helge Bahmann, and Magnus Själander. RVSDG: An Intermediate Representation for Optimizing Compilers. Association for Computing Machinery (ACM) 19(6):1-28, 2020 true https://pretalx.com/euroscipy-2024/talk/U3EMKF/ https://pretalx.com/euroscipy-2024/talk/U3EMKF/feedback/ Room 6 Talk (15 mins + Q&A) 2024-08-29T14:30:00+02:00 14:30 00:20 In this talk we're introducing a new tool to build conda packages. It has been adopted by the conda community and is being rolled out in the widely used conda-forge distribution. The new recipe format has been vetted in multiple Conda Enhancement Proposals (CEPs). We are going to introduce the exciting new features of rattler-build (reproducible builds, high speed build execution, etc.). Using some examples, we will then discuss how you can use rattler-build & conda-forge to build highly optimized packages with SIMD and CUDA support. We will also take a look at `cibuildwheel` and recent improvements in the PyPI space for CUDA. euroscipy-2024-50898-building-optimized-packages-for-conda-forge-and-pypi High Performance Computing Wolf VollprechtBas Zalmstra en This talk has two aims: introducing a new tool – rattler-build – and showing how rattler-build is used to build highly optimized packages for the widely used conda-forge distribution (which ships thousands of scientific Python packages). rattler-build has been developed from scratch to replace `conda-build`. It's a package build tool written in Rust, on top of the `rattler` libraries. We have undergone a meticulous standardization process through a number of Conda Enhancement Proposals. The result is a refined recipe format that removes a lot of the warts of conda-build: - proper YAML - no semantic comments or arbitrary Jinja allowed - integrates nicely with VSCode and other editors because we have a proper JSON schema - strictly defined behavior for multi-output recipes The result is that rattler-build is infinitely faster than conda-build at evaluating recipes. `rattler-build` also comes with perfect reproducibility for reproducible, bit-by-bit equivalent packages. In `conda-forge`, there has been recent innovation to ship more optimized packages. In the past, packages that wanted to make use of SIMD features in the CPU had to implement "dynamic dispatching" at runtime - at the cost of a larger package size. Since some time, `conda-forge` defines multiple "cpu-levels". These are defined for `sse`, `avx2`, `avx512` or `ARM Neon`. On the client-side the maximum CPU level is detected and the best available package is then installed. This opens the doors for highly optimized packages on `conda-forge` that support the latest CPU features. We will show how to use this in practice with rattler-build. For GPUs, `conda-forge` has supported different `CUDA` levels for a long time, and we'll look at how that is used as well. Lastly, we also take a look at PyPI. There are ongoing discussions on how to improve support for wheels with CUDA support. We are going to discuss how the (pre-)PEP works and synergy possibilities of `rattler-build` and `cibuildwheel`. false https://pretalx.com/euroscipy-2024/talk/JXB79J/ https://pretalx.com/euroscipy-2024/talk/JXB79J/feedback/ Room 6 Talk (25 mins + Q&A) 2024-08-29T15:30:00+02:00 15:30 00:30 Simulators play a crucial role in scientific research, but accurately determining their parameters to reproduce observed data remains a significant challenge. Classical parameter inference methods often struggle due to the stochastic or black-box nature of these simulators. Simulation-based inference (SBI) offers a solution by enabling Bayesian parameter inference for simulation-based models: It only requires simulated data as input and returns a posterior distribution over suitable model parameters, including uncertainty estimates and parameter interactions. In this talk, we introduce SBI and present [`sbi`](https://sbi-dev.github.io/sbi/), an open source library that serves as a central resource for SBI practitioners and researchers, offering state-of-the-art SBI algorithms, comprehensive documentation and tutorials. euroscipy-2024-50755-simulated-data-is-all-you-need-bayesian-parameter-inference-for-scientific-simulators-with-sbi Scientific Applications Jan Boelts (Teusen) en A central challenge with using simulators is identifying parameters that accurately reproduce observed data. When working with complex, possibly stochastic and black-box simulators, parameter inference often amounts to hand tuning, grid-searches or optimization for single best-fitting parameters. These approaches can be inefficient and brittle when working with high-dimensional and noisy simulators. The Bayesian parameter inference approach overcomes these challenges. It infers a conditional distribution over the simulator parameters given the observed data and a specified parameter prior encoding domain knowledge. The inferred posterior distribution identifies all suitable parameters while quantifying uncertainties and potential parameter correlations. However, classical approximate inference algorithms like e.g. Markov Chain Monte Carlo (MCMC) or variational inference usually cannot be applied to scientific simulators because they require access to the likelihood of the simulator. The core idea of simulation-based inference (SBI) is to enable Bayesian parameters inference for scientific simulators by requiring only simulated data. To that end, modern SBI methods use neural network-based conditional density estimation (e.g., normalizing flows, diffusion models) to learn to approximate the posterior solely from data. After training, the estimator can be applied to observed data to obtain an approximation to the desired posterior distribution. Many new SBI methods have been developed in the last years, enabling its applications to a wide range of simulators. However, bringing SBI into application can be challenging for practitioners because of a lack of reliable and approachable software tools. The [`sbi` library](https://sbi-dev.github.io/sbi/) aims to close this applicability gap: It implements state-of-the-art SBI algorithms, provides methods for the entire SBI workflow including accuracy checks and plotting, and comes with a comprehensive documentation and detailed tutorials. `sbi` is a community project with more than 50 contributors from across Europe. It originated as a research code base at the [University of Tübingen](https://www.mackelab.org/) and is now maintained by researchers from Tübingen and the [appliedAI TransferLab](https://transferlab.ai/). Contributions are very welcome and hackathons and workshops are organized regularly. false https://pretalx.com/euroscipy-2024/talk/893KBK/ https://pretalx.com/euroscipy-2024/talk/893KBK/feedback/ Room 6 Talk (25 mins + Q&A) 2024-08-29T16:00:00+02:00 16:00 00:30 AiiDA is a workflow manager with a strong focus on reproducibility through automated data provenance. In this talk we discuss what it means to have full “data provenance” for scientific workflows, the advantages it offers, but also the challenges it represents for new users and how we deal with them. euroscipy-2024-50921-reproducible-workflows-with-aiida-the-power-and-challenges-of-full-data-provenance Scientific Applications Marnik BercxXing Wang en AiiDA is a robust open-source Python package to help researchers automate, manage, persist, share, and reproduce complex workflows. A defining feature of AiiDA is the automatic recording of the calculations' history, or “provenance”, including relevant data inputs and outputs. This allows for designing detailed interfaces of processes and workflows, using advanced queries to look for relevant results or share data. This makes AiiDA particularly suitable for building a sustainable computational infrastructure for running high-throughput workflows and facilitates sharing data and provenance in a FAIR way for publication. On the other hand, writing workflows while keeping in mind the requirements of tracking the full provenance can be cumbersome for new users. Until very recently, running a new external (i.e. non-Python) code required developing a dedicated plugin, connecting processes in a workflow using advanced Python concepts. For high-throughput performance on HPC systems, AiiDA inherently depended on services that are not always trivial to install, such as a PostgreSQL database and a RabbitMQ message broker. In the past year, several improvements have been made to improve its usability, with a particular focus on getting new users up and running as quickly as possible. In this talk, we’ll start with a brief overview of AiiDA’s philosophy and core features as a workflow manager and discuss solutions to the challenges described above. The new plugin package `aiida-shell` makes running any shell executable easy, without the need to develop a custom plugin, while preserving basic provenance. Adding support for SQLite databases and making RabbitMQ optional allows users who don’t need high performance or scalability to run without the need to install and configure these services. Furthermore, a new WorkGraph feature provides a powerful framework for designing flexible node-based workflows with basic Python knowledge. This flexibility is essential to allow users to piece together a workflow for their scientific use case quickly. The WorkGraph also allows management and visualization of workflows in web browsers and Jupyter notebooks. false https://pretalx.com/euroscipy-2024/talk/HKVEXW/ https://pretalx.com/euroscipy-2024/talk/HKVEXW/feedback/ Room 5 Maintainer track 2024-08-29T11:00:00+02:00 11:00 00:45 NumPy 2 had some significant changes in its API and required many downstream libraries and users to adapt. One of the larger new features is that the new DType API is now public. This C-API allows more powerful user defined DTypes, for which the new StringDType is an example. In the first part, I will give a brief overview of this API. Since many downstream projects needed to adapt and publish new versions, in the second part I recap the current and past difficulties in transitioning to NumPy 2. This part of the session will be a forum for open discussion to gauge the challenges faced by users in making this transition. euroscipy-2024-50928-numpy-s-new-dtype-api-and-2-0-transition Scientific Applications Sebastian Berg en One of the new features of NumPy 2.0 is a new variable length StringDType. This DType was written using the new DType C API which had been only experimentally available previously. In this talk I will introduce the new concepts for creating user DTypes. What is the C API to construct such a new DType and what are the most important methods that need to be implemented? The StringDType and similar dtypes experiments in https://github.com/numpy/numpy-user-dtypes will serve as examples for this. In the second part, I will recap how long maintainers needed to release downstream libraries compatible with NumPy 2. Unfortunately, availability of downstream libraries may not be a good indicator of end-user difficulties, which are harder to predict. Undoubtedly, this transition is still ongoing in some parts of the ecosystem and this session is a good opportunity to discuss it. How hard was it to adapt to the Python API changes, the promotion changes due to NEP 50, and the requirement to recompile with NumPy 2? Discussing these will help us make future decisions about similar breaking changes. false https://pretalx.com/euroscipy-2024/talk/CDB9NG/ https://pretalx.com/euroscipy-2024/talk/CDB9NG/feedback/ Room 5 Maintainer track 2024-08-29T13:20:00+02:00 13:20 01:40 Scientific python libraries struggle with the existence of several array and dataframe providers. Many important libraries currently mainly support NumPy arrays or pandas dataframes. However, as library authors we wish to allow users to smoothly use other array provides and simplify for example the use of GPUs without the need for explicit use of cuda enabled libraries. This session will be split into three related discussions around efforts to tackle this situation: * Dispatching and backend selection discussion * Array API adoption progress and discussion * Dataframe compatibility layer discussion euroscipy-2024-53823-dispatching-backend-selection-and-compatibility-apis High Performance Computing Guillaume LemaitreJoris Van den BosscheTim HeadErik WelchMarco GorelliSebastian BergAditi JunejaStéfan van der Walt en ## Dispatching and Backend Selection Discussion In the first part, we would like to briefly review the successful [`NetworkX` backend selection](https://networkx.org/documentation/stable/reference/backends.html) and work towards a possible future dispatching project under the Scientific Python umbrella, [`spatch`](https://github.com/scientific-python/spatch). Many projects implement multiple dispatching based on types. Other projects have experimented with backend selection that goes beyond type dispatching and allows swapping in a different algorithm. `NetworkX` has both and also added features such as including installed backends into its documentation. And an older experiment for a dispatching system was `uarray`. Projects such as scikit-learn currently focus on a hybrid approach: dispatching via the array API where possible, otherwise using a backend selection system. Dispatching and backend selection is a complex field with various possible implementations. We would like to discuss requirements, specifically with a project like `scikit-image` in mind, and discuss how these requirements can be achieved. This session is meant to be an open discussion to push forward a new `spatch` library: an example implementation of such a dispatching system. ## Array API adoption progress and discussion In the second part, we will discuss adoption of [Array API](https://github.com/data-apis/array-api/) into libraries such as SciPy, and scikit-learn. How did the support develop in the past year and what issues remain. ## Dataframe compatibility layers Finally, similarly to the Array API, solutions such as [narwhals](https://github.com/narwhals-dev/narwhals) provide a compatibility layer for libraries working with dataframes. false https://pretalx.com/euroscipy-2024/talk/8MXPRW/ https://pretalx.com/euroscipy-2024/talk/8MXPRW/feedback/