0.9 pydata-paris-2024 2024-09-25 2024-09-27 3 00:05 https://pretalx.com https://pretalx.com/media/pydata-paris-2024/img/pydata-logo-final-dbg_z6kzPx6.png Europe/Paris Gaston Berger Keynote 2024-09-25T09:00:00+02:00 09:00 00:45 With increased ease of smaller "AI" models, better chips and on-device learning, is it possible now to build and train your own models for your own use? In this keynote, we'll explore learnings of small, medium and large-sized model personalization, but driven by yourself and for yourself. A walk through what's possible, what's not and what we should prioritize if we'd like AI & ML to be made for everyone. pydata-paris-2024-50117-keynote-diy-personalization-when-how-and-why-to-build-your-own-models Katharine Jarmul en false https://pretalx.com/pydata-paris-2024/talk/RDWBWQ/ https://pretalx.com/pydata-paris-2024/talk/RDWBWQ/feedback/ Gaston Berger Talk 2024-09-25T09:45:00+02:00 09:45 00:30 In the past few years, web-based engineering software has been steadily gaining momentum over traditional desktop-based applications. It represents a significant shift in how engineers access, collaborate, and utilize software tools for design, analysis, and simulation tasks. However, converting desktop-based applications to web applications presents considerable challenges, especially in translating the functionality of desktop interfaces to the web. It requires careful planning and design expertise to ensure intuitive navigation and responsiveness. JupyterLab provides a flexible, interactive environment for scientific computing. Despite its popularity among data scientists and researchers, the full potential of JupyterLab as a platform for building scientific web applications has yet to be realized. In this talk, we will explore how its modular architecture and extensive ecosystem facilitate the seamless integration of components for diverse functionalities: from rich user interfaces, accessibility, and real-time collaboration to cloud deployment options. To illustrate the platform's capabilities, we will demo JupyterCAD, a parametric 3D modeler built on top of JupyterLab components. pydata-paris-2024-48476-building-web-based-engineering-applications-with-jupyterlab-components Trung Le en false https://pretalx.com/pydata-paris-2024/talk/QQRKPG/ https://pretalx.com/pydata-paris-2024/talk/QQRKPG/feedback/ Gaston Berger Talk 2024-09-25T10:30:00+02:00 10:30 00:30 The Jupyter stack has undergone a significant transformation in recent years with the integration of collaborative editing features: users can now modify a shared document and see each other's changes in real time, with a user experience akin to that of Google Docs. The underlying technology uses a special data structure called Conflict-free Replicated Data Types (CRDTs), that automatically resolves conflicts when concurrent changes are made. This allows data to be distributed rather than centralized in a server, letting clients work as if data was local rather than remote. In this talk, we look at new possibilities that CRDTs can unlock, and how they are redefining Jupyter's architecture. Different use cases are presented: a suggestion system similar to Google Doc's, a chat system allowing collaboration with an AI agent, an execution model allowing full notebook state recovery, a collaborative widget model. We also look at the benefits of using CRDTs in JupyterLite, where users can interact without a server. This may be a great example of a distributed system where every user owns their data and shares them with their peers. pydata-paris-2024-48466-collaborative-editing-in-jupyter David Brochart en false https://pretalx.com/pydata-paris-2024/talk/HMMGLA/ https://pretalx.com/pydata-paris-2024/talk/HMMGLA/feedback/ Gaston Berger Talk 2024-09-25T11:05:00+02:00 11:05 00:30 Jupyter based environments are getting a lot of traction for teaching computing, programming, and data sciences. The narrative structure of notebooks has indeed proven its value for guiding each student at it's own pace to the discovery and understanding of new concepts or new idioms (e.g. how do I extract a column in pandas?). But then these new pieces of knowledge tend to quickly fade out and be forgotten. Indeed long term acquisition of knowledge and skills takes reinforcement by repetition. This is the foundation of many online learning platforms like [Webwork](https://en.wikipedia.org/wiki/WeBWorK) or [WIMS](https://fr.wikipedia.org/wiki/WIMS) that offer exercises with randomization and automatic feedback. And of popular "AI-powered" apps -- e.g. to learn foreign languages -- that use spaced repetition algorithms designed by educational and neuro sciences to deliver just the right amount of repetition. What if you could author such exercizes as notebooks, to benefit from everything that Jupyter can offer (think rich narratives, computations, visualization, interactions)? What if you could integrate such exercises right into your Jupyter based course? What if a learner could get personalized exercise recommandations based on their past learning records, without having to give away these sensitive pieces of information away? That's [Jupylates](https://gitlab.dsi.universite-paris-saclay.fr/jupyter/jupylates/) (work in progress). And thanks to the open source scientific stack, it's just a small Jupyter extension. pydata-paris-2024-48506-jupylates-spaced-repetition-for-teaching-with-jupyter Chiara MarmoNicolas M. Thiéry en In this talk, we will first briefly review our [usage of Jupyter for teaching at Université Paris-Saclay](https://jupyter.gitlab.dsi.universite-paris-saclay.fr/README.html), to motivate how we came to develop Jupylates. Then, after a quick demo, we will discuss its design, describe its current and upcoming features, and report on its usage in two large freshmen classes in 2023-2024. Time permitting, we will zoom out and deliver a brief tour of our Jupyter course infrastructure, advertising in particular [Travo](https://gitlab.com/travo-cr/travo/) for GitLab-based nbgrader-compatible assignment management. false Screenshot https://pretalx.com/pydata-paris-2024/talk/JAMSS3/ https://pretalx.com/pydata-paris-2024/talk/JAMSS3/feedback/ Gaston Berger Talk 2024-09-25T11:40:00+02:00 11:40 00:30 JupyterLite is a JupyterLab distribution that runs entirely in the web browser, backed by in-browser language kernels. With standard JupyterLab, where kernels run in separate processes and communicate with the client by message passing, JupyterLite uses kernels that run entirely in the browser, based on JavaScript and WebAssembly. This means JupyterLite deployments can be scaled to millions of users without the need for individual containers for each user session, only static files need to be served, which can be done with a simple web server like GitHub pages. This opens up new possibilities for large-scale deployments, eliminating the need for complex cloud computing infrastructure. JupyterLite is versatile and supports a wide range of languages, with the majority of its kernels implemented using Xeus, a C++ library for developing language-specific kernels. In conjunction with JupyterLite, we present Emscripten-forge, a conda/mamba based distribution for WebAssembly packages. Conda-forge is a community effort and a GitHub organization which contains repositories of conda recipes and thus provides conda packages for a wide range of software and platforms. However, targeting WebAssembly is not supported by conda-forge. Emscripten-forge addresses this gap by providing conda packages for WebAssembly, making it possible to create custom JupyterLite deployments with tailored conda environments containing the required kernels and packages. In this talk, we delve deep into the JupyterLite ecosystem, exploring its integration with Xeus Mamba and Emscripten-forge. We will demonstrate how this can be used to create sophisticated JupyterLite deployments with custom conda environments and give an outlook for future developments like R packages and runtime package resolution. pydata-paris-2024-48583-jupyterlite-emscripten-forge-xeus-and-mamba-the-computational-quartet-for-in-browser-interactive-computing Jeremy TuloupThorsten BeierIan Thomas en false https://pretalx.com/pydata-paris-2024/talk/LVSPQG/ https://pretalx.com/pydata-paris-2024/talk/LVSPQG/feedback/ Gaston Berger Talk 2024-09-25T13:50:00+02:00 13:50 00:30 Building scalable ETL pipelines and deploying them in the cloud can seem daunting. It shouldn't be. Leveraging proper technologies can make this process easy. We will discuss the whole process of developing a composable and scalable ETL pipeline centred around Dask that is fully built with Open Source tools and how we can deploy to the cloud. pydata-paris-2024-48671-building-large-scale-etl-pipelines-with-dask Patrick Hoefler en Running regular large-scale Python jobs on the cloud as part of production data pipelines is a common requirement. Building scalable ETL pipelines operating at scale and deploying them in the cloud can seem daunting. Managing different tools, building an infrastructure that can scale to large datasets and making sure that things run reliable has many complexities. Modern workflow orchestration systems like Prefect, Dagster, Airflow, etc. all work well for running jobs on a regular cadence. Dask fulfils the scalability requirement and storage formats like Delta Lake or Iceberg can both deal with large scale data. Dask is a library for distributed computing with Python and integrates tightly with pandas and other libraries from the PyData stack. It offers a DataFrame API that wraps pandas and thus offers an easy transition into the big data space. We will dive into recent Dask developments like a new shuffle algorithm, a query optimizer and other smaller changes and investigate how they improve scalability and performance to make Dask a great fit for large scale ETL pipelines. Gone are the days when Dask performed poorly on benchmarks, the new Dask DataFrame implementation can now beat Spark on the popular TPC-H benchmarks in cost and performance. We will illustrate how all those tools fit very nicely and how we can glue them together to create a composable ETL pipeline. This includes the infrastructure and architecture of the pipeline. We will look at a live example how our pipeline operates. We discuss how we can test locally and seamlessly switch deployment to the cloud. false https://pretalx.com/pydata-paris-2024/talk/JVGAEU/ https://pretalx.com/pydata-paris-2024/talk/JVGAEU/feedback/ Gaston Berger Talk 2024-09-25T14:25:00+02:00 14:25 00:30 Apache Arrow has become a de-facto standard for efficient in-memory columnar data representation. Beyond the standardized and language-independent columnar memory format for tabular data, the Apache Arrow project also has a growing set of supplementary specifications and language implementations. This talk will give an overview of the recent developments in the Apache Arrow ecosystem, including ADBC, nanoarrow, new data types, and the Arrow PyCapsule protocol. pydata-paris-2024-48539-the-expanding-apache-arrow-universe-standardizing-and-accelerating-tabular-data-access-and-interchange Joris Van den Bossche en According to the website, Apache Arrow is a multi-language toolbox for accelerated data interchange and in-memory processing. At its core is the Arrow Columnar Format, a language-independent columnar memory format for tabular data. But the Arrow project encompasses many things, including serialization, messaging and database specifications and a variety of language implementations. And this ecosystem of specifications and tools (and libraries adopting those) is constantly growing. After a brief recap of the Apache Arrow project to set the stage, this talk will give an overview of the recent developments in the Apache Arrow ecosystem: from additions to the core Columnar Format (such as the string view type), newer specifications (such as the ADBC database specification, additions to Flight SQL, and the Arrow PyCapsule protocol) and implementations (such as nanoarrow). false https://pretalx.com/pydata-paris-2024/talk/NNWFC3/ https://pretalx.com/pydata-paris-2024/talk/NNWFC3/feedback/ Gaston Berger Talk 2024-09-25T15:00:00+02:00 15:00 00:30 Polars is a dataframe library taking the world by storm. It is very runtime and memory efficient and comes with a clean and expressive API. Sometimes, however, the built-in API isn't enough. And that's where its killer feature comes in: plugins. You can extend Polars, and solve practically any problem. No prior Rust experience required, intermediate Python and programming experience required. By the end of the talk, you will know how to write your own Polars Plugin! This talk is aimed at data practitioners. pydata-paris-2024-46772-polars-plugins-how-you-yes-you-can-extend-polars /media/pydata-paris-2024/submissions/HHNHMJ/polar_plugin_small_iKtMvsX.jpg Marco Gorelli en Have you ever had the experience of needing to write a really custom function? Did you end up using a custom Python lambda function and waiting endlessly whilst your code executed? Let's learn how to put an end to that! This talk is aimed at advanced dataframe users who want to go beyond what Polars offers them. The structure will be: - 5 minutes motivation: example of a custom function which is painfully slow - 5 minutes prerequisites: what's Polars Series? What's a Chunked Array? What's an Arrow Array? What's a validity mask? - 5 minutes Rust basics: the bare minimum, and recommended resources to learn it properly - 5 minutes: how to translate Python lambda functions to Rust plugins: elementwise single-column example (take the absolute value of each element), then non-elementwise example (cumulative sum) - 5 minutes addressing the motivating example with a Polars plugin, show how much faster it became with only ~30 lines of Rust code - 5 minutes: Q&A By the end of the session, attendees will know how to write their own Polars Plugin, and how to go about learning any prerequisites they may currently lack (such as Rust). This talk is aimed at data practitioners who have experience with Python and data analysis (however, no prior Rust experience is required!). false https://pretalx.com/pydata-paris-2024/talk/HHNHMJ/ https://pretalx.com/pydata-paris-2024/talk/HHNHMJ/feedback/ Gaston Berger Keynote 2024-09-25T15:45:00+02:00 15:45 00:45 In this talk, we will go through everything open-source AI: the state of open-source AI, why it matters, the future of it and how you can get started with it. pydata-paris-2024-51836-keynote-open-source-ai-why-it-matters-and-how-to-get-started Merve Noyan en false https://pretalx.com/pydata-paris-2024/talk/J8GMY3/ https://pretalx.com/pydata-paris-2024/talk/J8GMY3/feedback/ Louis Armand 1 - Est Talk 2024-09-25T09:45:00+02:00 09:45 00:30 In the last year there hasn’t been a day that passed without us hearing about a new generative AI innovation that will enhance some aspect of our lives. On a number of tasks large probabilistic systems are now outperforming humans, or at least they do so “on average”. “On average” means most of the time, but in many real life scenarios “average” performance is not enough: we need correctness ALL of the time, for example when you ask the system to dial 911. In this talk we will explore the synergy between deterministic and probabilistic models to enhance the robustness and controllability of machine learning systems. Tailored for ML engineers, data scientists, and researchers, the presentation delves into the necessity of using both deterministic algorithms and probabilistic model types across various ML systems, from straightforward classification to advanced Generative AI models. You will learn about the unique advantages each paradigm offers and gain insights into how to most effectively combine them for optimal performance in real-world applications. I will walk you through my past and current experiences in working with simple and complex NLP models, and show you what kind of pitfalls, shortcuts, and tricks are possible to deliver models that are both competent and reliable. The session will be structured into a brief introduction to both model types, followed by case studies in classification and generative AI, concluding with a Q&A segment. pydata-paris-2024-48573-would-you-rely-on-chatgpt-to-dial-911-a-talk-on-balancing-determinism-and-probabilism-in-production-machine-learning-systems Nicolas Guenon des Mesnards en Objective and Outline: This talk addresses the often-overlooked need for integrating deterministic and probabilistic models in machine learning, which is crucial in complex production environments. We begin by defining deterministic and probabilistic models, highlighting their distinct roles in ML systems. The talk then showcases practical examples where the synergy of these models enhances system performance, focusing on classification and Generative AI models. Target Audience and Expected Background Knowledge: Intended for ML engineers, data scientists, and academic researchers, this presentation assumes familiarity with basic machine learning concepts and models. It's particularly beneficial for those involved in designing, implementing, or managing ML systems in production environments. Key Takeaways: - Understanding the strengths and limitations of deterministic and probabilistic models in ML. - Strategies for effectively combining these models in various ML systems. - Real-world examples demonstrating the improved robustness and controllability achieved through this integration. - Insights into future trends and potential developments in model integration. Time Breakdown: - Minutes 0-10: Introduction to deterministic and probabilistic models - Minutes 10-20: Synergies of approaches in real-world examples - Minutes 20-30: Applications for Generative AI models, including Q&A Additional Information: No prerequisites are required beyond a basic understanding of machine learning concepts. The presentation will be informative with a focus on practical applications, providing attendees with actionable knowledge and a deeper appreciation of model integration in ML systems. false Slides of presentation https://pretalx.com/pydata-paris-2024/talk/GT83UH/ https://pretalx.com/pydata-paris-2024/talk/GT83UH/feedback/ Louis Armand 1 - Est Talk 2024-09-25T10:30:00+02:00 10:30 00:30 Retrieval-augmented generation (RAG) has become a key application for large language models (LLMs), enhancing their responses with information from external databases. However, RAG systems are prone to errors, and their complexity has made evaluation a critical and challenging area. Various libraries (like RAGAS and TruLens) have introduced evaluation tools and metrics for RAGs, but these evaluations involve using one LLM to assess another, raising questions about their reliability. Our study examines the stability and usefulness of these evaluation methods across different datasets and domains, focusing on the effects of the choice of the evaluation LLM, query reformulation, and dataset characteristics on RAG performance. It also assesses the stability of the metrics on multiple runs of the evaluation and how metrics correlate with each other. The talk aims to guide users in selecting and interpreting LLM-based evaluations effectively. pydata-paris-2024-48503-evaluating-the-evaluator-rag-eval-libraries-under-the-loop Nour El MawassMaria Knorps en Retrieval-augmented generation (RAG) has emerged as a common technique to augment LLMs. It provides more founded answers based on relevant content from databases and documents, thus offering a cheap and convenient alternative to fine-tuning and re-training. However, a system that uses RAG can and will make mistakes, and the space of parameters one can tune in a RAG system is increasingly more complex. Evaluation has therefore understandably become a hot topic in the community. Many actors (e.g. RAGAS, TruLens, MLFlow) have developed libraries and RAG-specific metrics that allow for a faster, more principled and more easily scalable evaluation. This comes with a catch though: in a way, you're asking an LLM (albeit an admittedly high-grade one) to evaluate another LLM. To understand the usefulness and stability of RAG evaluation libraries, we have conducted an analysis of RAG evaluation results on two different datasets of corporate and technical documentation, representing different levels of technical difficulty. Manual evaluation is a time consuming process and we only provide a limited comparison of LLM-based results to manual ones. Instead, we dedicate the bulk of the analysis to more systematic ways of measuring how stable the results are. We study for instance the impact of the evaluator LLM choice and query reformulation on retrieval and generation in RAG, and re-run LLMs on both sides (RAG and evaluation) multiple times to assess the bounds of the metrics. We also study the correlation between different evaluation metrics and the impact of the dataset, its domain and its size. The goal of this talk is to provide the audience with the insights, caveats and better intuition they need when choosing, using and interpreting the results of LLM-based evaluation. Some prior knowledge in RAG is useful, but not required. false https://pretalx.com/pydata-paris-2024/talk/KLBGJA/ https://pretalx.com/pydata-paris-2024/talk/KLBGJA/feedback/ Louis Armand 1 - Est Talk 2024-09-25T11:05:00+02:00 11:05 00:30 Graph Retrieval Augmented Generation (Graph RAG) is emerging as a powerful addition to traditional vector search retrieval methods. Graphs are great at representing and storing heterogeneous and interconnected information in a structured manner, effortlessly capturing complex relationships and attributes across different data types. Using open weights LLMs removes the dependency on an external LLM provider while retaining complete control over the data flows and how the data is being shared and stored. In this talk, we construct and leverage the structured nature of graph databases, which organize data as nodes and relationships, to enhance the depth and contextuality of retrieved information to enhance RAG-based applications with open weights LLMs. We will show these capabilities with a demo. pydata-paris-2024-48288-enhancing-rag-based-apps-by-constructing-and-leveraging-knowledge-graphs-with-open-weights-llms Alonso Silva en The idea behind Retrieval Augmented Generation (RAG) applications is to provide Large Language Models (LLMs) with additional context at query time for answering the user’s question. Graph Retrieval Augmented Generation (Graph RAG) is emerging as a powerful addition to these traditional vector search retrieval methods. Graphs are great at representing and storing heterogeneous and interconnected information in a structured manner, effortlessly capturing complex relationships and attributes across diverse data types. In contrast, vector databases often struggle with such structured information, as their strength lies in handling unstructured data through high-dimensional vectors. Graph RAGs work as follows: - When a user asks a question, it first goes through an embedding model to calculate its vector representation. - The next step is to find the most relevant nodes in the database by comparing the cosine similarity of the embedding values of the user’s question and the relevant nodes in the database. - Once the relevant nodes are identified, the application is designed to retrieve additional information from the nodes themselves and also by traversing the relationships in the graph. - We can combine structured graph data with vector search through unstructured text to achieve the best of both worlds. - The context information from these databases is combined with the user question and additional instructions into a prompt that is passed to an LLM to generate the final answer, which is then sent to the user. Using open weights LLMs removes the dependency on an external LLM provider while retaining complete control over the data flows and how the data is being shared and stored. In this talk, we first construct a knowledge graph based on the provided documents with an open weights LLM. We then leverage the structured nature of graph databases, which organize data as nodes and relationships, to enhance the depth and contextuality of retrieved information to enhance RAG-based applications with open weights LLMs. We will show these capabilities with a demo. The tools used in the demo will be Kùzu + LanceDB + Llama.cpp + Outlines + LangChain. false https://pretalx.com/pydata-paris-2024/talk/AZNEHE/ https://pretalx.com/pydata-paris-2024/talk/AZNEHE/feedback/ Louis Armand 1 - Est Talk 2024-09-25T11:40:00+02:00 11:40 00:30 For some natural language processing (NLP) tasks, based on your production constraints, a simpler custom model can be a good contender to off-the-shelf large language models (LLMs), as long as you have enough qualitative data to build it. The stumbling block being how to obtain such data? Going over some practical cases, we will see how we can leverage the help of LLMs during this phase of an NLP project. How can it help us select the data to work on, or (pre)annotate it? Which model is suitable for which task? What are common pitfalls and where should you put your efforts and focus? pydata-paris-2024-48461-leveraging-llms-to-build-supervised-datasets-suitable-for-smaller-models Justine BEL-LETOILECérès Carton en Lately, large language models (LLMs) have appeared to make wonders for just about any natural language processing (NLP) task. But for many reasons, one may be unwilling or unable to use such models in production. Which is not a problem if the task can be tackled fairly well with a different kind of model, as long as one has enough qualitative training and testing data. But how to obtain said data? This is where LLMs come back into play. We will see how language models can help us create good datasets for different NLP tasks (classification, named entity recognition) with a fraction of the cost, time and pain it would normally require. From data selection to data annotation, we will go over some practical cases. We will try to answer some questions along the way: which model for which task, how much data do I need, and is this the ultimate dream or do you still need to put in a bit of effort? (spoiler alert: you do). false https://pretalx.com/pydata-paris-2024/talk/TBREDY/ https://pretalx.com/pydata-paris-2024/talk/TBREDY/feedback/ Louis Armand 1 - Est Talk 2024-09-25T13:50:00+02:00 13:50 00:30 Spatial Transcriptomics, named the method of the year by Nature in 2020, offers remarkable visuals of gene expression across tissues and organs, providing valuable insights into biological processes. This talk presents the Squidpy library for analyzing and visualizing spatial molecular data, including demonstrations of gene expression visualization in mouse brain tissue. pydata-paris-2024-48125-unveiling-new-maps-of-biology-with-squidpy Inácio Medeiros en Considered the method of the year by Nature in 2020, Spatial Transcriptomics has unveiled remarkable visuals of how genes are expressed across tissues and organs of living organisms. This breakthrough offers valuable insights to scientists, shedding light on the inner workings of biological processes such as brain function and cancer development. In this talk, we will explore how to build such visuals using Squidpy, a python library for the analysis and visualization of spatial molecular data. First, we will take a look at how Spatial Transcriptomics emerged inside bioinformatics and why it became the method of the year. Next, we will have a brief overview of the Squidpy library, as well as practical demonstrations showcasing how to visualize gene expression in mouse brain tissue. Lastly, we will discuss currently challenges on Spatial Transcriptomics. With this talk, you will gain a better understanding of spatial transcriptomics and will learn how to utilize the Squidpy library for visualizing gene expression in tissue samples. false https://pretalx.com/pydata-paris-2024/talk/MFRP9X/ https://pretalx.com/pydata-paris-2024/talk/MFRP9X/feedback/ Louis Armand 1 - Est Talk 2024-09-25T14:25:00+02:00 14:25 00:30 When scaling geoscience workloads to large datasets, many scientists and developers reach for Dask, a library for distributed computing that plugs seamlessly into Xarray and offers an Array API that wraps NumPy. Featuring a distributed environment capable of running your workload on large clusters, Dask promises to make it easy to scale from prototyping on your laptop to analyzing petabyte-scale datasets. Dask has been the de-facto standard for scaling geoscience, but it hasn’t entirely lived up to its promise of operating effortlessly at massive scale. This comes up in a few ways: - Correctly chunking your dataset has a significant impact on Dask’s ability to scale - Workers accidentally run out of memory due to: - Data being loaded too eagerly - Rechunking - Unmanaged memory Over the last few months, Dask has addressed many of those pains and continues to do so through: - Improvements to its scheduling algorithms - A faster and more memory-stable method for rechunking - First-of-its-kind logical optimization layer for a distributed array framework (ongoing) Join us as we dive into real-world geoscience workloads, exploring how Dask empowers scientists and developers to run their analyses at massive scale. Discover the impact of improvements made to Dask, ongoing challenges, and future plans for making it truly effortless to scale from your laptop to the cloud. pydata-paris-2024-48585-geoscience-at-massive-scale Hendrik Makait en false https://pretalx.com/pydata-paris-2024/talk/MXFAWK/ https://pretalx.com/pydata-paris-2024/talk/MXFAWK/feedback/ Louis Armand 1 - Est Talk 2024-09-25T15:00:00+02:00 15:00 00:30 Almost all modern CPU have a vector processing unit, making it possible to write faster code for a large category of problems, at the cost of portability - there a re many different instruction sets in the wild! The xsimd library makes it possible to write portable C++ code that targets different architectures and sub-architectures. The specialization choice can be made at compile-time or at runtime, using a provided dispatching mechanism. Intel, ARM, RiscV and Webassembly are supported, and the library has already been adopted by Xtensor, Pythran, Apache Arrow and Firefox. pydata-paris-2024-48435-xsimd-from-xtensor-to-firefox Serge « sans » Paille en From smartphone running on basic ARM chips to HPC station with the latest Intel CPUs, most modern CPU have a dedicated vector processing unit, making SIMD computing an optimization angle for many applications. Unfortunately, each chipset has its own instruction set, with different flavor across CPU families and within the same CPU family. Performance at the price of portability. The xsimd template library abstracts those difference to provide a common high-level API, bridging the gap between the different archtiectures and providing composite fallbacks when a one-to-one mapping with the ISA doesn't exist. It started as a vectorization backend for the xtensor library, has been adopted by Pythran, Krita, Apache Arrow and lately by Firefox to power its AI-based offline translation engine. It supports all Intel SSE variants, AVX, AVX2 and several AVX-512 extensions. ARM Neon with a few extensions, WebAssembly and RiscV are also supported target. In addition to these architectures, xsimd supports a special scalar target to have a similar API to target scalar and vector code, and an emulated target for testing purpose. This talk will present the xsimd API, showcase a few examples and discuss the difference with intrinsic-based programming and compiler auto vectorization approaches. false https://pretalx.com/pydata-paris-2024/talk/JBUSWA/ https://pretalx.com/pydata-paris-2024/talk/JBUSWA/feedback/ Louis Armand 2 - Ouest Talk 2024-09-25T09:45:00+02:00 09:45 00:30 Python 3.12 introduced a new low-impact monitoring API with [PEP669](https://peps.python.org/pep-0669/), which can be used to implement far faster debuggers than ever before. This talk covers the main advantages of this API and how you can use it to develop small tools. pydata-paris-2024-48274-python-3-12-s-new-monitoring-and-debugging-api Johannes Bechberger en Python long lagged a good monitoring and profiling API. It had only the simplistic sys.settrace API, which had a high overhead and couldn't be configured appropriately. The new API, released in October 2023, will change this by offering a proper fine-grained and well-designed monitoring API while also making the commonly used operations fast. This talk will give you an introduction to the new API and its design major design decisions and show you how you can use it to write a simple debugger from scratch. false https://pretalx.com/pydata-paris-2024/talk/ZQTCPC/ https://pretalx.com/pydata-paris-2024/talk/ZQTCPC/feedback/ Louis Armand 2 - Ouest Talk 2024-09-25T10:30:00+02:00 10:30 00:30 Are you looking for a high performance visualization component for the web? Need to filter, sort, pivot, and aggregate static/streaming data in realtime? Daunted by the massive JS ecosystem? In this talk, we’ll build a high performance web frontend using the open source library Perspective. pydata-paris-2024-47912-high-performance-data-visualization-for-the-web /media/pydata-paris-2024/submissions/3NDDQC/Screenshot_2024-03-26_at_15.36.39_FMYuVMl.png Tim Paine en The Python ecosystem has ample supply of both web development frameworks, and data visualization components. But despite the maturity of the ecosystem, few datavisualization tools are capable of dealing with large amounts of streaming data. Even fewer are able to perform live aggregations, sorting, and filtering on top of this data. In this talk, we will put together a simple but full-featured website using [Perspective](https://perspective.finos.org). Perspective is an open source interactive analytics and data visualization component, which is especially well-suited for large and/or streaming datasets. It is written in C++ and Rust with bindings to both Python and WebAssembly, making it ideal for data-intensive applications. It comes with a variety of visualization plugins, including a datagrid and various charts. Additionally, it comes with a Jupyter widget, which allows developers to iterate quickly with a clear pathway to their production website. We will start with a simple [FastAPI](https://fastapi.tiangolo.com)-based website and some static data. In a few lines of code, we will have the website up and running. Next, we will demonstrate some of the core features of Perspective - pivoting, sorting, filtering, the various visualization plugins, cross-filtering (using one table as a filter on other tables), and computed columns. After this, we will pull in some streaming data and show how the functionality of Perspective demonstrated updates in realtime alongside the data. Finally, we'll crank the speed of updates to the limit. By the end of this talk, the audience will know how to use Perspective and how to incorporate it into their own applications for both static and streaming data, either as a simple but high performance datagrid or as a full featured set of interconnected visualization components. false https://pretalx.com/pydata-paris-2024/talk/3NDDQC/ https://pretalx.com/pydata-paris-2024/talk/3NDDQC/feedback/ Louis Armand 2 - Ouest Talk 2024-09-25T11:05:00+02:00 11:05 00:30 Embark on a journey to explore how Quarto Dashboard can enhance the narrative of your analysis from your Jupyter Notebook. This talk will show how to create cool interactive charts and graphs that bring your data to life, by using Quarto - an open-source scientific and technical publishing system. Learn how to make your data communications more engaging and dynamic using Quarto Dashboard. Practical examples and simple explanations will guide you through the process, making it easy to understand and apply to your projects. pydata-paris-2024-48417-elevating-data-stories-exploring-quarto-dashboard-for-impactful-and-visual-communication Christophe Dervieux en **Content of the talk:** In this session, we'll first learn briefly about Quarto and how it could be used with Jupyter. Then, we'll explore the features of Quarto Dashboard and how it can transform your linear data analysis into dynamic visual stories. You'll see how easy it is to customize your data visualizations, organize results in advanced layouts, add interactive elements, and create eye-catching dashboards. The presentation will show an example of a dashboard created with Quarto and Jupyter. By the end of the talk, you'll have the tools and knowledge to elevate your data presentations and make them more impactful and engaging. **Target Audience:** This talk is for anyone interested in improving their data visualization skills, from beginners to experienced professionals. Whether you're a data scientist, analyst, educator, or just curious about data storytelling, you'll find value in learning how Quarto Dashboard can enhance data presentations and analysis simply and practically. No prior Quarto knowledge is needed. **Key Takeaways**: - Learn how Quarto works effectively with Jupyter Notebooks to create Dashboard output to publish - Discover the customization options available in Quarto Dashboard - Understand how Quarto Dashboard can enhance your data storytelling and analysis processes false https://pretalx.com/pydata-paris-2024/talk/XKGQXH/ https://pretalx.com/pydata-paris-2024/talk/XKGQXH/feedback/ Louis Armand 2 - Ouest Talk 2024-09-25T11:40:00+02:00 11:40 00:30 Many Python frameworks are suitable for creating basic dashboards or prototypes but struggle with more complex ones. Taking lessons from the JavaScript community, the experts on building UI’s, we created a new framework called Solara. Solara scales to much more complex apps and compute-intensive dashboards. Built on the Jupyter stack, Solara apps and its reusable components run in the Jupyter notebook and on its own production quality server based on Starlette/FastAPI. Solara has a declarative API that is designed for dynamic and complex UIs yet is easy to write. Reactive variables power our state management, which automatically triggers rerenders. Our component-centric architecture stimulates code reusability, and hot reloading promotes efficient workflows. With our rich set of UI and data-focused components, Solara spans the entire spectrum from rapid prototyping to robust, complex dashboards. pydata-paris-2024-48469-solara-pure-python-web-apps-beyond-prototypes-and-dashboards Maarten BreddelsIisakki Rotko en Solara bridges the gap between many Python web frameworks suitable for prototypes and full-stack web apps. We have looked at web frameworks in the JavaScript community, such as ReactJS, SolidJS, VueJS, and learned how they scale to larger apps. Declarative UIs and a dedicated state management system are the core of the solutions. We have built Solara on the Jupyter stack (ipywidgets) to avoid re-inventing the wheel. It also means we can reuse many of the widgets already created (such as anywidget based libraries). Planned outline of my talk: * Introduction/context * The problem: what is the problem we are trying to solve * Introducing Solara: general high-level overview of Solara * Solara in depth: go over the problems solara solves, and demo them with code (live demo in Jupyter Notebook) * Solara use cases: how is Solara currently used, real-life examples to show it * Quality focus: highlight the testing and typing support * Conclude The key takeaway: With Solara, you can create complex dashboards in Python beyond what you previously thought possible. Solara docs: https://solara.dev/ Solara GitHub: https://github.com/widgetti/solara/ I've given popular talks at SciPy before: https://www.youtube.com/watch?v=VtchVpoSdoQ - Dashboarding with Jupyter Notebooks, Voila and Widgets | SciPy 2019 | M. Breddels and M. Renou https://www.youtube.com/watch?v=hOKa8klJPyo - Interactive 3D Visualization in Jupyter | SciPy 2018 | Maarten Breddels https://www.youtube.com/watch?v=ELtjRdPT8is - Vaex: Out of Core Dataframes for Python | SciPy 2019 | Maarten Breddels false Slides online https://pretalx.com/pydata-paris-2024/talk/DFAEZ7/ https://pretalx.com/pydata-paris-2024/talk/DFAEZ7/feedback/ Louis Armand 2 - Ouest Talk 2024-09-25T13:50:00+02:00 13:50 00:30 Pixi goes further than existing conda-based package managers in many ways: * From scratch implemented in Rust and ships as a single binary * Integrates a new SAT solver called resolvo * Supports lockfiles like `poetry` / `yarn` / `cargo` do * Cross-platform task system (simple `bash`-like syntax) * Interoperability with PyPI packages by integrating `uv` * It's 100% open-source with a permissive licence We’re looking forward to take a deep-dive together into what conda and PyPI packages are and how we are seamlessly integrating the two worlds in pixi. We will show you how you can easily setup your new project using just one configuration file and always have a reproducible setup in your pocket. Which means that it will always run the same for your contributors, user and CI machine ( no more "but it worked on my machine!" ). Using pixi's powerful cross-platform task system you can replace your `Makefile` and a ton of developer documentation with just `pixi run task`! We’ll also look at benchmarks and explain more about the difference between the conda and pypi ecosystems. This talk is for everyone who ever dealt with [dependency hell](https://prefix.dev/blog/launching_pixi). More information about Pixi: https://pixi.sh https://prefix.dev https://github.com/prefix-dev/pixi pydata-paris-2024-48448-bridging-the-worlds-pixi-reimplements-pip-and-conda-in-rust Nichita Morcotilo en false https://pretalx.com/pydata-paris-2024/talk/WTXRDB/ https://pretalx.com/pydata-paris-2024/talk/WTXRDB/feedback/ Louis Armand 2 - Ouest Talk 2024-09-25T14:25:00+02:00 14:25 00:30 In this presentation, we introduce Mamba 2.0, the latest version of the multi-platform, language-agnostic package manager that has garnered significant adoption within the scientific open-source community for its speed and efficiency. pydata-paris-2024-48524-unveiling-mamba-2-0-the-future-of-fast-package-management Julien JerphanionJohan Mabille en Initially developed as a Python wrapper around OpenSUSE's libsolv SAT solver, Mamba has evolved into a full-fledged native package manager, compatible with Conda packages. Its recent integration into the conda-forge project and conda itself has enabled them to scale beyond previous limits. In this presentation, we will unveil the key enhancements and innovations in Mamba 2.0. We will explore the extensive refactoring process that transformed Mamba into a flexible framework, beyond the popular command-line utility, and delve into the main components of the core library. We will also highlight new key features such as support for OCI registry and the ability to integrate different SAT solvers. The talk will conclude with a glimpse into the future of Mamba, with a particular focus on software supply chain security. This includes plans for package signing and the implementation of a Software Bill Of Materials (SBOM), further solidifying Mamba's commitment to secure and reliable package management. false https://pretalx.com/pydata-paris-2024/talk/837ZAH/ https://pretalx.com/pydata-paris-2024/talk/837ZAH/feedback/ Louis Armand 2 - Ouest Talk 2024-09-25T15:00:00+02:00 15:00 00:30 Did you know that all top PyPI packages declare their 3rd party dependencies? In contrast, only about 53% of scientific projects do the same. The question arises: How can we reproduce Python-based scientific experiments if we're unaware of the necessary libraries for our environment? In this talk, we delve into the Python packaging ecosystem and employ a data-driven approach to analyze the structure and reproducibility of packages. We compare two distinct groups of Python packages: the most popular ones on PyPI, which we anticipate to adhere more closely to best practices, and a selection from biomedical experiments. Through our analysis, we uncover common development patterns in Python projects and utilize our open-source library, FawltyDeps, to identify undeclared dependencies and assess the reproducibility of these projects. This discussion is especially valuable for enthusiasts of clean Python code, as well as for data scientists and engineers eager to adopt best practices and enhance reproducibility. Attendees will depart with actionable insights on enhancing the transparency and reliability of their Python projects, thereby advancing the cause of reproducible scientific research. pydata-paris-2024-48525-on-the-structure-and-reproducibility-of-python-packages-data-crunch Maria KnorpsZhihan Zhang en Python has emerged as one of the most popular programming languages, with an extensive ecosystem of packages and projects catering to diverse domains. However, the structure and reproducibility of Python projects vary significantly, posing challenges for developers and researchers alike. In this talk, we perform a meta-analysis of Python packages in two different domains to gain a better understanding of wide-spread patterns and quantify the impact of missing or mis-configured dependencies on reproducibility. The FawltyDeps team was driven by a nagging question: How reproducible are Python packages and is there a discernible difference between top packages and those developed for scientific research? So we did what any data-loving person would do - we collected data and analyzed it using the core logic of FawltyDeps and indispensable Jupyter notebooks, e.g. biomedical data analysis. Our analysis includes packages from biomedical domain, drawing on the article by Sheeba and Mietchen, and for top PyPI packages, we used public BigQuery data. Our goal is to compare and contrast data-related projects with the top packages to unearth insights into the significant ways these two groups differ, both internally and in relation to each other, in terms of structure and dependency configuration. For instance, we found that nearly 20% of data science projects store all code and notebook files in the main directory, indicating a trend. This latter sample showcases data science projects crafted by developers of varying expertise levels, in which we anticipate greater variability in both structure and reproducibility. Outline: Introduction (0-2’): We introduce the speakers and provide an overview of the talk's objectives. Reproducibility of Python Projects (3-10’): We delve into the importance of reproducibility in Python projects and discuss common challenges faced by developers. Methodology and Data Collection (11-15’): We outline the experimental methodology employed, including the selection of biomedical experiments and PyPI packages for analysis and the different types of analysis done on those packages. Results and Insights (16-23’): We present the findings of our analysis, highlighting common patterns in Python project development and assessing their reproducibility using FawltyDeps. Key Takeaways (24-25’): We emphasize the key lessons learned from our analysis, particularly the importance of structured project development and dependency declarations. Question and Answer Session (26-30’): false Another result - a comparison of usage of files where 3rd party dependencies are declared: requirements.txt, setup.py, setup.cfg, pyproject.toml and their combinations. https://pretalx.com/pydata-paris-2024/talk/9RZ8MH/ https://pretalx.com/pydata-paris-2024/talk/9RZ8MH/feedback/ Gaston Berger Keynote 2024-09-26T09:00:00+02:00 09:00 00:45 In the rapidly evolving landscape of Artificial Intelligence (AI), open source and openness AI have emerged as crucial factors in fostering innovation, transparency, and accountability. Mistral AI's release of the open-weight Mistral 7B model has sparked significant adoption and demand, highlighting the importance of open-source and customization in building AI applications. This talk focuses on the Mistral AI model landscape, the benefits of open-source and customization, and the opportunities for building AI applications using Mistral models. pydata-paris-2024-51865-keynote-building-with-mistral Sophia Yang en false https://pretalx.com/pydata-paris-2024/talk/U7CM7L/ https://pretalx.com/pydata-paris-2024/talk/U7CM7L/feedback/ Gaston Berger Talk 2024-09-26T09:45:00+02:00 09:45 00:30 One of the more mundane tasks in the business analytics world is to measure KPIs: averages, sums, ratios, etc. Typically, these are measured period over period, to see how they trend. If you're a data analyst, you've likely been asked to debug/explain a metric, because a stakeholder wants to understand why a number has changed. This topic isn't well grounded theory, and the answers we come up with can be lacklustre. In this talk, we discuss solutions to this very common topic. We will look at a methodology we have developed at Carbonfact, and the opensource Python tool we are sharing. pydata-paris-2024-47653-unpack-business-metrics-to-explain-their-evolution /media/pydata-paris-2024/submissions/Y9S73X/create-kpi-dashboard_CvndtgL.jpeg Max Halford en This talk is for all the people who perform analytics in their daywork. The goal of the talk is to add a tool to your belt. You'll leave the talk with a robust framework for answering questions from your stakeholders. For instance, you'll be able to attribute the change in a metric to the drivers of said metric, down to the decimal point. The presentation will present the (simple) maths of the methodology we propose. It's a simplified version of the Blinder-Oaxaca decomposition framework. We will then present the Python toolbox we are open-sourcing, along with a tutorial. Finally, we will discuss what are the best ways to share such methods in the current data tooling landscape (e.g. with the rise of DuckDB and Polars). The talk will be given by Max Halford, who is Head of Data at Carbonfact. Max has already given a talk at PyData Amsterdam, and regularly gives talks at different venues. false https://pretalx.com/pydata-paris-2024/talk/Y9S73X/ https://pretalx.com/pydata-paris-2024/talk/Y9S73X/feedback/ Gaston Berger Talk 2024-09-26T10:30:00+02:00 10:30 00:30 Understanding the effectiveness of various marketing channels is crucial to maximise the return on investment (ROI). However, the limitation of third-party cookies and an ever-growing focus on privacy make it difficult to rely on basic analytics. This talk discusses a pioneering project where a **Bayesian model** was employed to assess the marketing media mix effectiveness of **[WeRoad](https://www.weroad.it/)**, the fastest-growing Italian tour operator. The Bayesian approach allows for the incorporation of prior knowledge, seamlessly updating it with new data to provide robust, actionable insights. This project leveraged a Bayesian model to unravel the complex interactions between marketing channels such as online ads, social media, and promotions. We'll dive deep into how the Bayesian model was designed, discussing how we provided the AI system with expert knowledge, and presenting how **delays and saturation** were modelled. We will also tackle aspects of the technical implementation, discussing how Python, **[PyMC](https://www.pymc.io/welcome.html)**, and Streamlit provided us with the all the tools we needed to develop an effective, efficient, and user-friendly system. Attendees will walk away with: - A simple understanding of the Bayesian approach and why it matters. - Concrete examples of the transformative impact on WeRoad's marketing strategy. - A blueprint to harness predictive models in their business strategies. pydata-paris-2024-47950-is-your-marketing-effective-let-bayes-decide Emanuele Fabbiani en false https://pretalx.com/pydata-paris-2024/talk/ZLJZRT/ https://pretalx.com/pydata-paris-2024/talk/ZLJZRT/feedback/ Gaston Berger Talk 2024-09-26T11:05:00+02:00 11:05 00:30 Discover metalearners, a cutting-edge Python library designed for Causal Inference with particularly flexible and user-friendly MetaLearner implementations. metalearners leverages the power of conventional Machine Learning estimators and molds them into causal treatment effect estimators. This talk is targeted towards data professionals with some Python and Machine Learning competences, guiding them to optimizing interventions such as 'Which potential customers should receive a voucher to optimally allocate a voucher budget?' or 'Which patients should receive which medical treatment?' based on causal interpretations. pydata-paris-2024-48241-catering-causal-inference-an-introduction-to-metalearners-a-flexible-metalearner-library-in-python Kevin KleinFrancesc Martí Escofet en We will introduce metalearners, a particularly flexible and user-friendly MetaLearner implementation for Causal Inference in the Python ecosystem. Through this session, we aim to demonstrate how metalearners, coupled with **standard ML estimators**, can produce **robust causal models** for **treatment effects** in light of optimizing interventional **decision making** or policy learning. Unlike existing implementations, it provides a high degree of flexibility: using elaborate pandas and polars features, convenient investigation of base estimators or the ability of reusing pre-trained models are some of its advantages. ### Background In many scenarios in which we intervene with the real world, e.g. administering drugs to patients or distributing vouchers to potential clients, correlation in observational data and prediction models just don't quite do the trick. Conceptually and methodologically, Causal Inference offers the appropriate tools for such scenarios. Yet, practice hasn't quite kept up with the developments on paper. In particular, implementations of MetaLearners - a relatively recent and exciting development in the field of Causal Inference - currently still lack user-friendly, flexible and efficient implementations. metalearners seeks to fill this gap. ### Agenda - Why care about treatment effect estimation? (5') - Introduction to MetaLearners for treatment effect estimation (5') - Some shortcomings with existing libraries (5') - Demo (10') - Q&A (5') ### Target Audience This talk aims at Data Scientists, ML engineers, AI researchers, and Analysts with some Python familiarity, an understanding of basic Machine Learning concepts and at least some curiosity about causality. ### Key Takeaways - Insights into merging Causal Inference with standard Machine Learning models - Understanding of metalearners' role in flexible and scalable MetaLearner implementations false https://pretalx.com/pydata-paris-2024/talk/LTFWZK/ https://pretalx.com/pydata-paris-2024/talk/LTFWZK/feedback/ Gaston Berger Talk 2024-09-26T11:40:00+02:00 11:40 00:30 In their seminal paper "Why propensity scores should not be used for matching," King and Nielsen (2019) highlighted the shortcomings of Propensity Score Matching (PSM). Despite these concerns, PSM remains prevalent in mitigating selection bias across numerous retrospective medical studies each year and continues to be endorsed by health authorities. Guidelines to mitigating these issues have been proposed, but many researchers encounter difficulties in both adhering to these guidelines and in thoroughly documenting the entire process. In this presentation, I show the inherent variability in outcomes resulting from the commonly accepted validation condition of Standardized Mean Difference (SMD) below 10%. This variability can significantly impact treatment comparisons, potentially leading to misleading conclusions. To address this issue, I introduce A2A, a novel metric computed on a task specifically designed for the problem at hand. By integrating A2A with SMD, our approach substantially reduces the variability of predicted Average Treatment Effects (ATE) by up to 90% across validated matching techniques. These findings collectively enhance the reliability of PSM outcomes and lay the groundwork for a comprehensive automated bias correction procedure. Additionally, to facilitate seamless adoption across programming languages, I have integrated these methods into "popmatch," a Python package that not only incorporates these techniques but also offers a convenient Python interface for R's MatchIt methods. pydata-paris-2024-48522-dreadful-frailties-in-propensity-score-matching-and-how-to-fix-them Alexandre Abraham en This presentation aims at raising awareness on the pressing challenge of evaluating bias correction methods, which suffers from a lack of ground truth for validation. While validation typically relies on expert opinion,we propose a novel metric and lay the groundwork for a comprehensive automated pipeline. This pipeline aims to offer consistent results across experiments, even if not optimal, thereby addressing the current variability in outcomes. Primarily geared towards practitioners in the medical field who utilize propensity score matching, this talk offers practical examples and steers clear of excessive theoretical discussions. Nevertheless, the core message is applicable to various bias correction methods across different domains. The presentation aims to identify the problem, provides a rudimentary solution ready for refinement, and advocates for public engagement in the project. The talk is structured as follows: - Identifying the Problem: Using real and synthetic datasets, we demonstrate how different population matching techniques can produce varying results, highlighting the need for a robust evaluation method. - Introduction of Ground Truth in Bias Correction: We demonstrate how existing data can be leveraged to construct artificial problems with known solutions, offering a benchmark for evaluation. - Crafting a Metric: We present A2A, a novel metric designed to evaluate bias correction methods based on the repeated creation of synthetic problems. We employ this metric to gain a deeper understanding of the phenomenon. - Future Perspectives: We discuss the limitations of the proposed metric and avenues for refinement. Additionally, we contextualize this endeavor within the broader framework of building a fully automated bias correction engine, highlighting remaining tasks to encourage contributions. false https://pretalx.com/pydata-paris-2024/talk/VJWXQL/ https://pretalx.com/pydata-paris-2024/talk/VJWXQL/feedback/ Gaston Berger Talk 2024-09-26T13:50:00+02:00 13:50 00:30 MAPIE (Model Agnostic Prediction Interval Estimator) is your go-to solution for managing uncertainties and risks in machine learning models. This Python library, nestled within scikit-learn-contrib, offers a way to calculate prediction intervals with controlled coverage rates for regression, classification, and even time series analysis. But it doesn't stop there - MAPIE can also be used to handle more complex tasks like multi-label classification and semantic segmentation in computer vision, ensuring probabilistic guarantees on crucial metrics like recall and precision. MAPIE can be integrated with any model - whether it's scikit-learn, TensorFlow, or PyTorch. Join us as we delve into the world of conformal predictions and how to quickly manage your uncertainties using MAPIE. Link to Github: https://github.com/scikit-learn-contrib/MAPIE pydata-paris-2024-48546-boosting-ai-reliability-uncertainty-quantification-with-mapie Louis LacombeThibault CordierValentin Laurent en This talk introduces MAPIE, an open-source Python library designed to quantify uncertainties and control risks in machine learning models. Learn how to compute conformal prediction intervals and control risks in various tasks such as regression, classification, time series, and even complex tasks like multi-label classification and semantic segmentation. We will begin by discussing the importance of uncertainty quantification and risk control in machine learning models. Then, we will dive into the key features of MAPIE, including: 1. Computing conformal prediction intervals for regression, classification, and time series tasks with guaranteed marginal coverage rates. 2. Controlling risks for complex tasks such as multi-label classification and semantic segmentation in computer vision. 3. Wrapping any machine learning model (scikit-learn, TensorFlow, PyTorch, etc.) with a scikit-learn-compatible wrapper for uncertainty quantification and risk control. Throughout the talk, we will demonstrate MAPIE's capabilities with practical examples and code snippets. Attendees will learn how to apply MAPIE to their own models, ensuring more reliable and robust predictions. This talk targets data scientists, machine learning engineers, and researchers with a basic understanding of machine learning concepts. Familiarity with scikit-learn and other popular machine learning libraries is helpful but not required. By the end of the talk, attendees will have gained valuable insights into uncertainty quantification and risk control, as well as hands-on experience using MAPIE to bring uncertainty quantification to their machine learning models. Approximate time breakdown: 0-3 minutes: Why perform uncertainty quantification? 3-15 minutes: Quick dive into the conformal prediction intervals for regression and classification. 15-20 minutes: Overview of a real-world application + Interactive demo of MAPIE. 20-30 minutes: Q&A and discussion. false https://pretalx.com/pydata-paris-2024/talk/QQSZCJ/ https://pretalx.com/pydata-paris-2024/talk/QQSZCJ/feedback/ Gaston Berger Talk 2024-09-26T14:25:00+02:00 14:25 00:30 Adaptive prediction intervals, which represent prediction uncertainty, are crucial for practitioners involved in decision-making. Having an adaptivity feature is challenging yet essential, as an uncertainty measure must reflect the model's confidence for each observation. Attendees will learn about state-of-the-art algorithms for constructing adaptive prediction intervals, which is an active area of research. pydata-paris-2024-48161-adaptive-prediction-intervals Andro Sabashvili en Proper prediction uncertainty estimation is critical, particularly when data-driven decision-making is in place. Point predictions alone are insufficient as they only represent the conditional expectation of a target variable. Therefore, to make well-informed decisions, one has to quantify the variance of probability distribution associated with a point prediction. Prediction intervals provide this valuable information. Conformal prediction is a powerful distribution-free framework allowing to construct prediction intervals with valid marginal coverage. However, in real-life scenarios, predictive model performance is not homogeneous; there are regions in the input space where it performs better, leading to smaller prediction uncertainties that should be mapped to narrower prediction intervals. This capability to adapt interval lengths based on local performance is called adaptivity. In this talk, targeted towards data scientists who develop and deploy predictive models, we review and benchmark algorithms for constructing adaptive prediction intervals. The key takeaway for the audience will be learning about the algorithms for building adaptive prediction intervals and understanding how to apply them in their own work. false https://pretalx.com/pydata-paris-2024/talk/DKHKHG/ https://pretalx.com/pydata-paris-2024/talk/DKHKHG/feedback/ Gaston Berger Talk 2024-09-26T15:00:00+02:00 15:00 00:30 In this talk, we provide an update on the latest `scikit-learn` features that have been implemented in versions 1.4 and 1.5. We will particularly discuss the following features: - the metadata routing API allowing to pass metadata around estimators; - the `TunedThresholdClassifierCV` allowing to tuned operational decision through custom metric; - better support for categorical features and missing values; - interoperability of array and dataframe. pydata-paris-2024-48353-an-update-on-the-latest-scikit-learn-features Guillaume LemaitreStefanie Senger en In this talk, we present a couple of important novelties available `scikit-learn` since version 1.4. First, we discuss the new metadata routing feature and its API. We start by giving a concrete example of metadata through the usage of `sample_weight` and `groups`. These parameter were previously available in many estimators, metrics, and evaluation tools. However, in old `scikit-learn` version, this feature came with shortcomings that we examine. Then, we show how the new metadata routing API solves these challenges with examples that involve some nested-ness. Concretely, we show how to enable and use this routing on a use-case involving cross-validation, `scikit-learn` pipeline, and grid-search. We further give another concrete example where metadata routing unlock an important feature: it is now possible to optimize a classifier decision threshold using a dedicated business metric. While this was not possible in the past, it is now possible in `scikit-learn` thanks to metadata routing coupled with a new meta-estimator called the `TunedThresholdClassifierCV` that allows optimizing the classification decision threshold. Then, we present some native handling of categorical and missing data in some estimators (i.e., gradient-boosting and random forest) that leverage `pandas` data types. Finally, we present some advancements regarding interoperability with different input data. Notably, we show the premise of adopting the so-called Array-API (<https://data-apis.org/array-api/>). This work brings `scikit-learn` to the GPU processing land and allows for improved performance for some algorithms. We showcase an example using a typical machine learning pipeline. In addition, we present the recent support for another dataframe library, `polars`. false https://pretalx.com/pydata-paris-2024/talk/7JJDF8/ https://pretalx.com/pydata-paris-2024/talk/7JJDF8/feedback/ Gaston Berger Keynote 2024-09-26T15:45:00+02:00 15:45 00:45 Machine Learning practitioners build predictive models from "noisy" data resulting in uncertain predictions. But what does "noise" mean in a machine learning context? pydata-paris-2024-48697-keynote-handling-predictive-uncertainty-in-machine-learning Olivier Grisel en Over years of developing scikit-learn and exchanging with both applied data scientists and academics, I have progressively refined my understanding on topics such as the fundamental sources of "noisy" data, how to evaluate and improve the design of predictive models with respect to ranking power and probabilistic calibration, how to handle predictive uncertainty and turn probabilistic forecasts into optimal decisions with respect to an application-specific utility function. The goal of this keynote to share what I learned with you and hopefully help you reflect on how to quantify the value of predictions and better use scikit-learn and other predictive modeling tools. Slides: https://docs.google.com/presentation/d/1EBCSCDQ3nTPaKZGx9ZLWXfvkD1Y-ODo9j_ETAnx5zLQ/edit?usp=sharing false https://pretalx.com/pydata-paris-2024/talk/7VYUGP/ https://pretalx.com/pydata-paris-2024/talk/7VYUGP/feedback/ Louis Armand 1 - Est Talk 2024-09-26T09:45:00+02:00 09:45 00:30 NetworkX is arguably the most popular graph analytics library available today, but one of its greatest strengths - the pure-python implementation - is also possibly its biggest weakness. If you're a seasoned data scientists or a new student of the fascinating field of graph analytics, you're probably familiar with NetworkX and interested in how to make this extremely easy-to-use library powerful enough to handle realistically large graph workflows that often exceed the limitations of its pure-python implementation. This talk will describe a relatively new capability of NetworkX; support for accelerated backends, and how they can benefit NetworkX users by allowing it to finally be both easy to use and fast. Through the use of backends, NetworkX can also be incorporated into workflows that take advantage of similar accelerators, such as Accelerated Pandas (cudf.pandas), to finally make these easy to use solutions scale to larger problems. Attend this talk to learn about how you can leverage the various backends available to NetworkX today to seamlessly run graph analytics on GPUs, use GraphBLAS implementations, and more, all without leaving the comfort and convenience of the most popular graph analytics library available. pydata-paris-2024-48537-fast-networkx-and-how-accelerated-backends-are-changing-graph-analytics Erik WelchRick Ratzel en NetworkX is arguably the most popular graph analytics library available today; it's extremely easy to install and use, supports a huge assortment of popular analytics, and has excellent support via its community and documentation, and the number of PyPI downloads - typically around 47 million every month - support this. But despite these benefits, many users are forced to move on to another graph analytics library when the limitations of NetworkX's pure-python implementation prevents them from using it for larger workloads. Will NetworkX users ever be able to have the ubiquity and ease-of-use they've grown accustomed to without the limitations of a pure-python implementation? The answer is yes! Graph analytics that are fast and scalable but still easy to use and well-documented are possible simply by using NetworkX with an accelerated backend. The backends available to NetworkX today enable support for running analytics on GPUs, leveraging linear algebra through GraphBLAS, and more, all without requiring users to change their existing NetworkX code. This changes the state of graph analytics by making it accessible to more users and more use cases than ever before. This talk will describe how you can take advantage of the several NetworkX backends available today as well as how you can incorporate your own custom backend to handle graph-related problems that simply weren't feasible before. Examples, benchmark comparisons, and descriptions of the underlying dispatching mechanism that makes alternate backends possible will be provided to illustrate how ease of use and performance can finally coexist in NetworkX. This talk will also show how NetworkX with backend acceleration can be incorporated into workflows that take advantage of similar accelerators, such as Accelerated Pandas (cudf.pandas), to finally make these easy to use solutions scale to larger problems. Attend this talk to hear just how easy it is to use NetworkX to unlock the hidden secrets in your large, real-world data that only fast graph analytics can reveal. Detailed background knowledge of NetworkX and graph analytics is not required. An approximate time breakdown is as follows: 5 mins - Brief introduction to graph analytics and NetworkX, why it's the most popular graph analytics library, and what its limitations are. 15 mins - How do accelerated backends work, what's available, and how can they make NetworkX work for workloads both big and small? 5 mins - NetworkX's bright future, and how you can help! 5 mins - Q&A false https://pretalx.com/pydata-paris-2024/talk/A7VSKN/ https://pretalx.com/pydata-paris-2024/talk/A7VSKN/feedback/ Louis Armand 1 - Est Talk 2024-09-26T10:30:00+02:00 10:30 00:30 Transformers are everywhere: NLP, Computer Vision, sound generation and even protein-folding. Why not in forecasting? After all, what ChatGPT does is predicting the next word. Why this architecture isn't state-of-the-art in the time series domain? In this talk, you will understand **how Amazon Chronos and Salesforece's Moirai transformer-based forecasting models work**, the **datasets used to train them** and **how to evaluate them** to see if they are a good fit for your use-case. pydata-paris-2024-48386-foundational-models-for-time-series-forecasting-are-we-there-yet Luca BaggiGabriele Orlandi en This is a pragmatic talk for Generative AI and ML practitioners alike, eager to understand how researchers managed to **adapt transformer-based architectures to forecasting problems**, as well as how to **evaluate new models for specific use-cases**. The first part of the talk explains how researchers **discretised time series into a finite "dictionary"** and the **theoretical limitations** of this approach, such as how to deal with different sampling frequencies during training. We will then make a parallel with Large Language Models (LLMs) scaling laws to evaluate the data strategies used to train such universal forecasting models. In other words, we will ask the question **whether we have enough publicly available time-series data to train foundational models**, and how this can **affect such models' evaluation**. Finally, we will display how we attempted to **benchmark those models against a robust baseline of models**, and where our experiment is compared to the publicly available results. No specific prior knowledge is required for attendance. Though forecasting practitioners stand to gain the most from this talk, every practitioner in machine learning or generative AI can follow along and draw the key conclusions. **Outline** Minutes 1-3. Problem statement: challenges in adapting transformers to the time-series domain. Minutes 3-10. How Chronos and Moirai are implemented. Minutes 10-15. Do we have enough data to train universal forecasters? Minutes 15-25. Lessons learnt from evaluating transformer-based models for a specific use-case. Minutes 25-30. Wrap up and Q&A. false https://pretalx.com/pydata-paris-2024/talk/E7LHZX/ https://pretalx.com/pydata-paris-2024/talk/E7LHZX/feedback/ Louis Armand 1 - Est Talk 2024-09-26T11:05:00+02:00 11:05 00:30 sktime is a widely used scikit-learn compatible library for learning with time series. sktime is easily extensible by anyone, and interoperable with the pydata/numfocus stack. This talk presents progress, challenges, and newest features off the press, in extending the sktime framework to deep learning and foundation models. Recent progress in generative AI and deep learning is leading to an ever-exploding number of popular “next generation AI” models for time series tasks like forecasting, classification, segmentation. Particular challenges of the new AI ecosystem are inconsistent formal interfaces, different deep learning backends, vendor specific APIs and architectures which do not match sklearn-like patterns well – every practitioner who has tried to use at least two such models at the same time (outside sktime) will have their individual painful memories. We show how sktime brings its unified interface architecture for time series modelling to the brave new AI frontier, using novel design patterns building on ideas from hugging face and scikit-learn, to provide modular, extensible building blocks with a simple specification language. pydata-paris-2024-48521-sktime-python-toolbox-for-time-series-next-generation-ai-deep-learning-and-foundation-models Franz KiralyBenedikt Heidrich en sktime’s technical mission is to integrate the ecosystem of time series modelling behind a simple, unified, scikit-learn like first-order modelling language. After a years-long journey of integration and stringent architecture, sktime provides unified interfaces for hundreds of algorithms for tasks such as time series forecasting or classification, which can be used as building blocks in a highly composable interface. sktime compatible algorithms – 1st party in the package itself, or 3rd party in private extensions – follow consistent, exchangeable interfaces, which are kept robust and extensible through strong API checks and testing tools. While sktime, as a framework and marketplace, is nowadays a go-to place for the “canon” of statistical, econometric, ML models, a new, rapidly evolving “wild west” has emerged, with a growing wave of time series foundation models being released, and renewed interest in deep learning based algorithms. In the talk, we will discuss specific challenges that need to be addressed, on the framework layer, estimator specific dependency management layer, related to availability of algorithms, and project governance: - The need to cover pre-training, fine-tuning, which implies handling of pre-trained weights - e.g., from hugging face - as well as extending sklearn-like interfaces to fine-tuning style update or extending task models such as to global forecasting - The need to cover back-end abstractions for data containers, deep learning frameworks, and possibly vendor specific APIs; implying interfaces and layers usually not found in scikit-learn-like libraries; and patterns that require closer integration across packages - As an explicitly neutral, non-commercial library, dealing with extreme commercial pressures and related behavioural patterns, e.g., companies wanting to establish “their algorithm” and trying to nudge users towards – and lock them in to – proprietary APIs rather than framework APIs - The interaction with open research and data ecosystems, as foundation models require closer integration with data corpora for training and tuning We think the above pose a new set fundamental challenges, similar to the initial gap that sktime had to bridge, between a world without a comprehensive framework for time series ML, and the current state in which sktime exists (and in which multiple projects take inspiration from its design ideas). Our talk outlines new features, current roadmap, and partial progress towards building a framework in which next generation algorithms do not only coexist with classical ML, but become fully interoperable, comparable, and composable. sktime is developed by an open community, with aims of ecosystem integration in a commercially neutral, charitable space. We welcome contributions or donations, and seek to provides opportunity for anyone worldwide. false https://pretalx.com/pydata-paris-2024/talk/KZ8ZPA/ https://pretalx.com/pydata-paris-2024/talk/KZ8ZPA/feedback/ Louis Armand 1 - Est Talk 2024-09-26T11:40:00+02:00 11:40 00:30 Markov chain Monte Carlo (MCMC) methods, a class of iterative algorithms that allow sampling almost arbitrary probability distributions, have become increasingly popular and accessible to statisticians and scientists. But they run into difficulties when applied to multimodal probability distributions. These occur, for example, in Bayesian data analysis, when multiple regions in the parameter space explain the data equally well or when some parameters are redundant. Inaccurate sampling then results in incomplete and misleading parameter estimates. Markov chain Monte Carlo (MCMC) methods, a very popular class of iterative algorithms that allow sampling almost arbitrary probability distributions, run into difficulties when applied to multimodal probability distributions. These occur, for example, in Bayesian data analysis, when multiple regions in the parameter space explain the data equally well or when some parameters are redundant. In this talk, intended for data scientists and statisticians with basic knowledge of MCMC and probabilistic programming, I present Chainsail, an open-source web service written entirely in Python. It implements Replica Exchange, an advanced MCMC method designed specifically to improve sampling of multimodal distributions. Chainsail makes this algorithm easily accessible to users of probabilistic programming libraries by automatically tuning important parameters and exploiting easy on-demand provisioning of the (increased) computing resources necessary for running Replica Exchange. pydata-paris-2024-48504-chainsail-facilitating-sampling-of-multimodal-probability-distributions Simeon Carstens en The presentation starts with a small refresher on MCMC and an introduction to the Replica Exchange algorithm. A second part will focus on the web service architecture, and finally, I will show examples of how Chainsail improves MCMC sampling. The audience will learn about a common pitfall when doing Bayesian data analysis and how an algorithm originally developed in the field of solid state physics can help. Furthermore, the architecture part of the presentation shows an example of how to set up a complex, multi-component web service using Kubernetes and other popular software. false https://pretalx.com/pydata-paris-2024/talk/AN7PX3/ https://pretalx.com/pydata-paris-2024/talk/AN7PX3/feedback/ Louis Armand 1 - Est Talk 2024-09-26T13:50:00+02:00 13:50 00:30 Open Source Software, the backbone of today’s digital infrastructure, must be sustainable for the long-term. Qureshi and Fang (2011) find that motivating, engaging, and retaining new contributors is what makes open source projects sustainable. Yet, as Steinmacher, et al. (2015) identifies, first-time open source contributors often lack timely answers to questions, newcomer orientation, mentors, and clear documentation. Moreover, since the term was first coined in 1998, open source lags far behind other technical domains in participant diversity. Trinkenreich, et al. (2022) reports that only about 5% of projects were reported to have women as core developers, and women authored less than 5% of pull requests, but had similar or even higher rates of pull request acceptances to men. So, how can we achieve more diversity in open source communities and projects? Bloomberg’s Women in Technology (BWIT) community, Open Source Program Office (OSPO), and Corporate Philanthropy team collaborated with NumFOCUS to develop a volunteer incentive model that aligns business value, philanthropic impact, and individual technical growth. Through it, participating Bloomberg engineers were given the opportunity to convert their hours spent contributing to the pandas open source project into a charitable donation to a non-profit of their choice. The presenters will discuss how we wove together differing viewpoints: non-profit foundation and for-profit corporation, corporate philanthropy and engineers, first-time contributors and core devs. They will showcase why and how we converted technical contributions into charitable dollars, the difference this community-building model had in terms of creating a diverse and sustained group of new open source contributors, and the viability of extending this to other open source projects and corporate partners to contribute to the long-term sustainability of open source—thereby demonstrating the true convergence of tech and social impact. NOTE: [1] Qureshi, I, and Fang, Y. "Socialization in open source software projects: A growth mixture modeling approach." 2011. [2] Steinmacher, I., et al. "Social barriers faced by newcomers placing their first contribution in open source software projects." 2015. [3] Trinkenreich, B., et al. "Women’s participation in open source software: A survey of the literature." 2022. pydata-paris-2024-48494-open-source-sustainability-philanthropy-building-contributor-communities Alyssa WrightDevpriya Dave en Built on collaborative communities, open source is critical infrastructure on which our digital world depends. It's important that me+we are represented in the communities sustaining open source. Bloomberg joined forces with the NumFOCUS Foundation in 2023 to pilot a new approach to open source sustainability to address the lack of diversity in—and drop-off rate of—new open source contributors to the pandas project. The result is a framework for sustainable open source contribution that open source projects can scale for growth and companies can scale for business and philanthropic value. This new volunteer incentive model leads the way in breaking down boundaries between philanthropy and technology. This series is leading the way in breaking down boundaries between philanthropic and technology alignment and impact for public good. This informative talk describes the structure of the series and how it helps to reimagine both the definition of philanthropy and how D&I leaders can raise the bar for technical excellence. 1. Welcome and initial remarks (5 mins) a. The importance of open source to Bloomberg b. Intro to Bloomberg’s pandas Sustainability Series 2. How Bloomberg sustains Open Source though philanthropy (5 mins) a. Bloomberg’s Open Source Program Office (OSPO) and its Sustainability pillar b. Intro to the firm’s Open Source Dollars For Your Hours program i. Turning hours spent making open source contributions into charitable donations to the non-profit of engineers’ choice 3. Impact of the Bloomberg Women in Technology (BWIT) community (5 mins) a. Intro to BWIT Philanthropy b. BWIT’s partnership with OSPO to build collaborative communities in the open source ecosystem c. Reimagining the definition of philanthropy and demonstrating how women leaders in engineering continue to raise the bar for technical excellence 4. Pandas Sustainability Series (5 min) a. How we made it happen b. High notes c. What we learned 5. Create your own sustainability series (5 mins) a. How you can do it too 6. Audience Q&A (5 mins) false https://pretalx.com/pydata-paris-2024/talk/YVFJFF/ https://pretalx.com/pydata-paris-2024/talk/YVFJFF/feedback/ Louis Armand 1 - Est Talk 2024-09-26T14:25:00+02:00 14:25 00:30 The EU Commission is likely to vote on the Cyber Resilience Act (CRA) later this year. In this talk we will look at the timeline for the new legislation, any critical discussions happening around implementation and most importantly, the new responsibilities outlined by the CRA. We’ll also discuss what the PSF is doing for CPython and for PyPI and what each of us in the Python ecosystem might want to do to get ready for a new era of increased certainty – and liability – around security. pydata-paris-2024-48432-counting-down-for-cra-updates-and-expectations Cheuk Ting Ho en The EU Commission is likely to vote on the Cyber Resilience Act (CRA) later this year. The CRA is an ambitious step towards protecting consumers from software security issues by creating a new list of responsibilities for software developers and providers. The Act also creates a new category of actor known as an “Open Source Steward” which we think makes important allowances for public open source repositories like CPython and Python Package Index (PyPI.) Once the dust settles, everyone who makes software will need to consider the CRA’s mandates in their security roadmaps. In this talk we will look at the timeline for the new legislation, any critical discussions happening around implementation and most importantly, the new responsibilities outlined by the CRA. We’ll also discuss what the PSF is doing for CPython and for PyPI and what each of us in the Python ecosystem might want to do to get ready for a new era of increased certainty – and liability – around security. ## Target audience Developers and maintainers whose project or product may be affected by the CRA. European legislation won’t just affect the European market, it will affect the software industry and the open source community globally as it is very hard to segregate one project or product from the EU market. So, this is for everyone in the Python community who shares their code with the world. ## Goal To educate the general public about CRA - how it can affect us and how to get ready for it. We also want to provide more information for the Python community about what has been done by the PSF regarding the CRA to reassure them that the Python community is aware and getting prepared for the CRA. ## Outline - Introduction - what is CRA - Highlight of the CRA text as of now - How we may be affected - Getting ready - What has PSF done for cyber security - What has PSF done for CRA specifically - What is PSF planning to do to support the community - Call for action - How to be prepared as an individual - How to be prepared if you are maintaining a project - How to support PSF to be ready for CRA false https://pretalx.com/pydata-paris-2024/talk/QMZRTS/ https://pretalx.com/pydata-paris-2024/talk/QMZRTS/feedback/ Louis Armand 1 - Est Talk 2024-09-26T15:00:00+02:00 15:00 00:30 Rising concerns over IT's carbon footprint necessitate tools that gauge and mitigate these impacts. This session introduces CodeCarbon, an open-source tool that estimates computing's carbon emissions by measuring energy use across hardware components. Aimed at AI researchers and data scientists, CodeCarbon provides actionable insights into the environmental costs of computational projects, supporting efforts towards sustainability without requiring deep technical expertise. This talk from the main contributors of Code Carbon will cover the environmental impact of IT, the possibilities to estimate it and a demo of CodeCarbon. pydata-paris-2024-48184-track-your-code-s-c02-emissions-with-code-carbon Luis BlancheBenoît Courty en **Background**: The IT sector's environmental impact is significant and growing, driven by the surge in data processing and computational demands, particularly within the realms of AI and data science. Addressing this requires tools that not only measure but also help manage and reduce these impacts. **Introducing CodeCarbon**: CodeCarbon is a Python package that offers a straightforward method for estimating the carbon emissions of computing tasks. It calculates energy consumption (GPU, CPU, RAM) and translates this into carbon emissions based on the regional carbon intensity where the computation occurs. It's designed for both technical and non-technical users, with a command line interface and easy integration into existing code bases. **Session Objectives**: - *Environmental Impact Awareness*: Highlight the urgency of acknowledging and acting on the carbon footprint associated with IT projects. - *Tool Demonstration*: Walk through CodeCarbon's features, including setup, integration into different environments (command line, Jupyter notebook, and direct code integration), and how to utilize the dashboard for visualizing emissions data. - *Broad Applicability*: Emphasize CodeCarbon's utility beyond AI and data science, demonstrating its relevance for any computational work seeking to minimize environmental impact. - *Community Engagement* : Discuss how developers and researchers can contribute to CodeCarbon, enhancing its accuracy, usability, and impact. - *Takeaway* : Participants will gain a thorough understanding of how to leverage CodeCarbon to make informed decisions about the environmental implications of their work. The session underscores the importance of measurement in environmental sustainability, echoing Niels Bohr: "Nothing exists until it is measured." By quantifying the carbon footprint of computational tasks, CodeCarbon empowers the community to drive change towards more sustainable practices in technology. This talk aims to bridge the gap between awareness and action, providing the PyData community with the tools to actively reduce the carbon footprint of their computational endeavors. false https://pretalx.com/pydata-paris-2024/talk/TK9TKL/ https://pretalx.com/pydata-paris-2024/talk/TK9TKL/feedback/ Louis Armand 2 - Ouest Talk 2024-09-26T09:45:00+02:00 09:45 00:30 The astronomical community has built a good amount of software to visualize and analyze the images obtained with the James Webb Space Telescope (JWST). In this talk, I will present the open-source Python package Jdaviz. I will show you how to visualize publicly available JWST images and build the pretty color images that we have all seen in the media. Half the talk will be an introduction to JWST and Jdaviz and half will be a hands on session on a cloud platform (you will only need to create an account) or on your own machine (the package is available on PyPI). pydata-paris-2024-48230-color-composite-images-from-the-james-webb-space-telescope Camilla PacificiJesse Averbukh en Do you like the pretty images in those press releases from the James Webb Space Telescope (JWST)? Would you like to learn how to make one yourself using open source software? Yes? Then please come to my talk with your laptop and I will show you how to do it! JWST has been observing the Universe for about 2 years now, sending us spectacular images of planets, nebulae, stars, and galaxies. These images are taken in filters at different wavelengths and they need to be combined to produce the stunning color images we see in the press releases. There is various software available to do this. In this talk, I will present an open-source Python package called Jdaviz (J-D-A-viz) which stands for JWST Data Analysis and Visualization. Jdaviz works in the Jupyter environment and the users can interact with it using the interface or using Python commands from a Jupyter Notebook. No need to be a scientist or astronomer to follow this presentation. If you are familiar with Jupyter Lab, the concept of taking images through filters, and the concept of scaling the range and contrast of an image, you will enjoy the ride. The presentation will be light weighted and half of it will be a hands-on session. If you like the idea, I will be happy to use 30 minutes of PyData to: - give an introduction to JWST and astronomical images; 5 minutes - give an introduction to the Python open-source package Jdaviz; 5 minutes - be your guide to make a color-composite image using JWST data on a cloud platform (you will need to create an account) or on your own machine (the package is available on PyPI); 15 minutes - answer any questions you might have; 5 minutes false https://pretalx.com/pydata-paris-2024/talk/SNLQWH/ https://pretalx.com/pydata-paris-2024/talk/SNLQWH/feedback/ Louis Armand 2 - Ouest Talk 2024-09-26T10:30:00+02:00 10:30 00:30 [Aladin](https://aladin.cds.unistra.fr/aladin.gml) allows to visualize images of the sky or planetary surfaces just as an astronomical "openstreetmap" app. The view can be panned and explored interactively. In the [ipyaladin widget](https://github.com/cds-astro/ipyaladin) -- that brings Aladin in the Jupyter Notebook environnement -- these abilities are extended with a python API. The users can send astronomical data in standard formats back and forth the viewer and their Python code. Such data can be images of the sky in different wavelengths, but also tabular data, complex shapes that characterize telescope observation regions, or even special sky features (such as probability region for the provenance of a gravitational event). With these already existing features, and current work we are doing with the new development framework `anywidget`, `ipyaladin` is really close to a version 1.0.0. It is already used in its beta version in different experimental science platforms, for example in the ESCAPE [European Science Cluster of Astronomy & Particle Physics](https://projectescape.eu/) project and in the experimental [SKA](https://www.skao.int/en/science-users) (Square Kilometre Array, a telescope for radio astronomy) analysis platform. In this presentation, we will share our feedback on the development of a widget thanks to `anywidget` compared to the bare `ipywidget` framework. And we will demonstrate the functionalities of the widget through scientific use cases. pydata-paris-2024-48587-visualization-of-the-sky-in-notebooks-the-ipyaladin-widget-extension Matthieu BaumannManon Marchand en We write in the abstract that `ipyaladin` is close to its version 1.0.0, but our aim would be to present this release for the PyData event in September. - audience: scientific public of amateur astronomers / widget developers - background knowledge: none - outline: 1. problematic of visualizing spherical surfaces 2. what we want to show are huge datasets (windowing of the printed data in the field of view) 3. how to provide a Python API for this viewer for the scientific community 4. the widget development process 5. (bonus: beautiful images of the sky) false https://pretalx.com/pydata-paris-2024/talk/CVA9XR/ https://pretalx.com/pydata-paris-2024/talk/CVA9XR/feedback/ Louis Armand 2 - Ouest Talk 2024-09-26T11:05:00+02:00 11:05 00:30 The video is available here: https://www.youtube.com/watch?v=yn1bR-BVfn8&list=PLGVZCDnMOq0pKya8gksd00ennKuyoH7v7&index=37 This talk introduces [DataLab](https://datalab-platform.com), a unique open-source platform for signal and image processing, seamlessly integrating scientific and industrial applications. The main objective of this talk is to show how [DataLab](https://datalab-platform.com) may be used as a complementary tool alongside with Jupyter notebooks or an IDE (e.g., [Spyder](https://www.spyder-ide.org/)), and how it can be extended with custom Python scripts or applications. pydata-paris-2024-47030-datalab-bridging-scientific-and-industrial-worlds-for-advanced-signal-and-image-processing /media/pydata-paris-2024/submissions/WTDVCC/DataLab-Overview_VbIPMdm.png Pierre Raybaut en [DataLab](https://datalab-platform.com) offers a practical approach to signal and image processing, tailored for the scientific community. This platform, grounded in Python and leveraging the rich ecosystem of scientific libraries (e.g. NumPy, SciPy, scikit-image, OpenCV, PlotPyStack), supports a range of research activities. This talk will explore the main use cases of [DataLab](https://datalab-platform.com): - Processing and visualizing your signals and images (use DataLab as generic processing software) - Prototyping your processing pipeline (direct communication with your IDE or notebook) - Debugging your processing application (direct communication with DataLab from your application) - Enhancing your application with processing features (use DataLab as a library, or as a companion to your application) ***Real-World Scientific Applications*** This segment will present how [DataLab](https://datalab-platform.com) is used in various scientific fields. We will explore examples like spectral data analysis in chemistry, measuring laser beam sizes in physics, or identifying biological structures in medical imaging. These instances will demonstrate [DataLab](https://datalab-platform.com)'s adaptability and utility in real research scenarios, as a rapid prototyping tool for signal and image processing as well as a customizable platform for data visualization. ***Extensibility and Automation in Scientific Research*** [DataLab](https://datalab-platform.com)'s extensible nature allows researchers to tailor it to their specific needs. This part of the talk will illustrate how custom plugins and macros can enhance research capabilities, while its automation features streamline repetitive tasks, improving efficiency in scientific workflows. ***Bridging Scientific Research with Industrial Processes*** A case study will be presented, illustrating [DataLab](https://datalab-platform.com)'s application in a scientific setting, where it also interfaces with industrial control systems. This example will showcase [DataLab](https://datalab-platform.com)'s versatility and its potential to facilitate data processing in cross-disciplinary projects. false https://pretalx.com/pydata-paris-2024/talk/WTDVCC/ https://pretalx.com/pydata-paris-2024/talk/WTDVCC/feedback/ Louis Armand 2 - Ouest Talk 2024-09-26T11:40:00+02:00 11:40 00:30 In this talk, we'll look into why Insee had to go beyond usual tools like JupyterHub. With data science growing, it has become important to have tools that are easy to use, can change as needed, and help people work together. The opensource software Onyxia brings a new answer by offering a user-friendly way to boost creativity in a data environment that uses massively containerization and object storage. pydata-paris-2024-48483-onyxia-a-user-centric-interface-for-data-scientists-in-the-cloud-age Comte FrédéricJoseph GarroneLino GalianaQuentin Gaudel en https://onyxia.sh is an opensource software developed by the French public service that aims at providing a state-of-the-art working environment (python, R) for data teams based on kubernetes and S3. In this talk, we will present : - why we think that containerisation and object storage is mandatory to make a scalable data platform. - why we developped Onyxia and our principles to make it sustainable - how we design it and how we are trying to create a community Last but not least, you may learn that you are eligible to use our free, open-innovation platform named SSPCloud. false https://pretalx.com/pydata-paris-2024/talk/Z989SU/ https://pretalx.com/pydata-paris-2024/talk/Z989SU/feedback/ Louis Armand 2 - Ouest Talk 2024-09-26T13:50:00+02:00 13:50 00:30 Retrieval is the process of searching for a given item (image, text, …) in a large database that are similar to one or more query items. A classical approach is to transform the database items and the query item into vectors (also called embeddings) with a trained model so that they can be compared via a distance metric. It has many applications in various fields, e.g. to build a visual recommendation system like Google Lens or a RAG (Retrieval Augmented Generation), a technique used to inject specific knowledge into LLMs depending on the query. Vector databases ease the management, serving and retrieval of the vectors in production and implement efficient indexes, to rapidly search through millions of vectors. They gained a lot of attention over the past year, due to the rise of LLMs and RAGs. Although people working with LLMs are increasingly familiar with the basic principles of vector databases, the finer details and nuances often remain obscure. This lack of clarity hinders the ability to make optimal use of these systems. In this talk, we will detail two examples of real-life projects (Deduplication of real estate adverts using the image embedding model DinoV2 and RAG for a medical company using the text embedding model Ada-2) and deep dive into retrieval and vector databases to demystify the key aspects and highlight the limitations: HSNW index, comparison of the providers, metadata filtering (the related plunge of performance when filtering too many nodes and how indexing partially helps it), partitioning, reciprocal rank fusion, the performance and limitations of the representations created by SOTA image and text embedding models, … pydata-paris-2024-47956-towards-a-deeper-understanding-of-retrieval-and-vector-databases Noé Achache en Retrieval is the process of searching for a given item (image, text, …) in a large database that are similar to one or more query items. A classical approach is to transform the database items and the query item into vectors (also called embeddings) with a trained model so that they can be compared via a distance metric. It has many applications in various fields, e.g. to build a visual recommendation system like Google Lens or a RAG (Retrieval Augmented Generation), a technique consisting of retrieving the documents the most similar to a given question, to then answer the question with an LLM given the retrieved documents. Vector databases ease the management, serving and retrieval of the vectors in production and implement efficient indexes, to rapidly search through millions of vectors. They gained a lot of attention over the past year, due to the rise of LLMs and RAGs. Although people working with LLMs are increasingly familiar with the basic principles of vector databases, the finer details and nuances often remain obscure. This lack of clarity hinders the ability to make optimal use of these systems. For instance which differences matter between the different providers? How mature metadata filtering is, when to use it and what are the limitations? What is partitioning and what is the best way to do it? Which functionalities beyond simple vector search can be helpful? … In this talk, we will clarify these grey areas to help better leverage these systems. More specifically, we will: - 1/ Introduce the main concepts: - How retrieval works - What are vector databases - What are the use cases. We will introduce two real-life projects we did using vector databases, which will allow us to provide practical examples through the talk - Deduplication of real estate adverts using the image embedding model DinoV2 - RAG for a medical company using the text embedding model Ada-2 - 2/ Highlight the pros and cons of the different providers, with a highlight on the differences between dedicated vector databases and more general-purpose databases. - 3/ Deep dive into the technicality of vector search: - How does the HSNW index work and how to configure it - The necessity of metadata filtering and the related limitations: mostly when filtering out more than 80-90% of the nodes the precision plunges. Hence indexing becomes necessary but it is still imperfect in complex cases. - The performance and limitations of the representations created by state-of-the-art image and text embedding models false https://pretalx.com/pydata-paris-2024/talk/WHXMTY/ https://pretalx.com/pydata-paris-2024/talk/WHXMTY/feedback/ Louis Armand 2 - Ouest Talk 2024-09-26T14:25:00+02:00 14:25 00:30 The MedTech industry is undergoing a revolutionary transformation with continuous innovations promising greater precision, efficiency, and accessibility. In particular oncology, a branch of medicine that focuses on cancer, will benefit immensely from these new technologies, which may enable clinicians to detect cancer earlier and increase chances of survival. Detecting cancerous cells in microscopic photography of cells (Whole Slide Images, aka WSIs) is usually done with segmentation algorithms, which neural networks (NNs) are very good at. While using ML and NNs for image segmentation is a fairly standard task with established solutions, doing it on WSIs is a different kettle of fish. Most training pipelines and systems have been designed for analytics, meaning huge columns of small individual datums. In the case of WSIs, a single image is so huge that its file can be up to dozens of gigabytes. To allow innovation in medical imaging with AI, we need efficient and affordable ways to store and process these WSIs at scale. pydata-paris-2024-48149-processing-medical-images-at-scale-on-the-cloud Guillaume Desforges en In this presentation, we look at the challenges encountered in processing large-scale medical images, particularly in oncology, where images can be as large as several gigabytes. We discuss the limitations faced by most companies in accessing supercomputing resources and the necessity of cloud-based solutions. The separation of storage and compute, coupled with the complexities of distributed computing, presents significant hurdles. We explore methods and tools to address these challenges, enabling data scientists to iterate and experiment efficiently. Drawing from our experience at Modus Create (formerly Tweag) working with Kaiko.ai, we will share insights into building an AI research platform tailored to the fight against cancer. * Introduction, Whole Slide Images and detecting cancer (5 min) * The challenge of training Neural Networks on Whole Slide Images on the cloud (5 min) * Building blocks for an AI Research Platform for oncology (10 min) * Future developments (5 min) * Q&A (5 min) This talk assumes minimal Python and Cloud knowledge. No specific medical knowledge is required. false https://pretalx.com/pydata-paris-2024/talk/QWXJTN/ https://pretalx.com/pydata-paris-2024/talk/QWXJTN/feedback/ Louis Armand 2 - Ouest Talk 2024-09-26T15:00:00+02:00 15:00 00:30 Scaling machine learning at large organizations like Renault Group presents unique challenges, in terms of scales, legal requirements, and diversity of use cases. Data scientists require streamlined workflows and automated processes to efficiently deploy models into production. We present an MLOps pipeline based on python Kubeflow and GCP Vertex AI API designed specifically for this purpose. It enables data scientists to focus on code development for pre-processing, training, evaluation, and prediction. This MLOPS pipeline is a cornerstone of the AI@Scale program, which aims to roll out AI across the Group. We choose a Python-first approach, allowing Data scientists to focus purely on writing preprocessing or ML oriented Python code, also allowing data retrieval through SQL queries. The pipeline addresses key questions such as prediction type (batch or API), model versioning, resource allocation, drift monitoring, and alert generation. It favors faster time to market with automated deployment and infrastructure management. Although we encountered pitfalls and design difficulties, that we will discuss during the presentation, this pipeline integrates with a CI/CD process, ensuring efficient and automated model deployment and serving. Finally, this MLOps solution empowers Renault data scientists to seamlessly translate innovative models into production, and smoothen the development of scalable, and impactful AI-driven solutions. pydata-paris-2024-48586-mlops-at-renault-group-a-generic-pipeline-for-scalable-deployment Alix Tiran-CappelloAlexandre Carton en Topic: This presentation delves into the development and implementation of a robust MLOps pipeline at Renault, designed to address the unique challenges of scaling machine learning deployments in large, complex organizations. Audience: This talk is primarily aimed at data scientists, ML engineers, and anyone interested in: • Learning about efficient MLOps implementation for large-scale deployments. • Utilizing Python for comprehensive ML workflows, including data retrieval with SQL. • Addressing key considerations for production-ready model deployment and management. Attendees will gain insights into: • Implementing an MLOps pipeline tailored for real-world use cases. • Efficiently automating machine learning workflows for production environments. • Utilizing Python code for all stages of model development and deployment. • Addressing key considerations for scalable ML deployments at large organizations. • Potential pitfalls in creating a custom MLOPS solutions. Background Knowledge: • Basic understanding of Machine Learning and Python programming is recommended. • Familiarity with MLOps concepts and CI/CD pipelines would be beneficial. false https://pretalx.com/pydata-paris-2024/talk/ZREEJS/ https://pretalx.com/pydata-paris-2024/talk/ZREEJS/feedback/