Pydantic is a data validation library for Python that has seen massive adoption over the last few years - it's used by major datascience and ML libraries like Spacy, Huggingface and jinja-ai - overall Pydantic is downloaded over 55m times a month!

In this talk Samuel Colvin, the creator of Pydantic will cover two subjects which have seen massive interest in recent years:

• How Pydantic can be used to prepare data for machine learning thereby saving time and avoiding errors
• The emergence of Rust as the go-to language for high performance python libraries - how this might go in the future, and the benefits and drawbacks of the trend

Writing maintainable data science code is a big topic, and different people have different opinions on the best ways to do it. Wouldn't it be nice if there was an opinionated framework to set some structure and help data scientists be more effective and ship their analysis and models to production faster?

In this workshop we present Kedro, an opinionated Python framework for creating reproducible, maintainable and modular data science code. We will also show how you can combine it with Polars, a new dataframe library backed by Arrow and Rust, for lightning fast data manipulation and exploratory data analysis.

The importance of enterprise architecture patterns is all well-known and applicable to varied types of tasks. Thinking about the architecture from the beginning of the journey is crucial to have a maintainable, therefore testable, and flexible code base. In We are going to explore the Ports and Adapters(Hexagonal) pattern by showing a simple web app using Repository, Unit of Work, and Services(Use Cases) patterns tied together with Dependency Injection. All those patterns are quite famous in other languages but they are relatively new for the Python ecosystem, which is a crucial missing part.

Understanding software architecture and how the data flows within software components is a vital step toward building and maintaining software systems. Architecture diagrams help enable this through digital graphical designs mixed with human-computer interaction. Furthermore, these visualizations not only help system architects, but also developers, project managers, and even customers. The complexity in designing them arises not only from the fact that such systems are an intangible conceptual entity, but also, most importantly, that they are ever-evolving.

While we are searching for life on Mars, our software diagrams remain manual and lifeless. Imagine a life where you update the code for your software, and the architecture view gets updated automatically and is ready to be interacted with. Let's use Graph Databases, pandas, and Python to add life to them and make them interactive.

Building a strong data science portfolio can be a daunting task, especially for those just starting out in the field.
In this talk, we will explore the essential elements of a successful data science portfolio, and provide actionable advice for building a portfolio that will make recruiters take notice.

First, we will discuss the importance of selecting the right projects for your portfolio. We'll share tips for identifying projects that demonstrate a range of skills and showcase your expertise in a specific area.

Next, we'll talk about how to present your work in a clear and compelling way, including how to structure your portfolio and which tools and platforms to use.

We will also discuss how to incorporate feedback from peers and mentors, as well as how to solicit feedback from potential employers. In addition, we'll cover best practices for maintaining and updating your portfolio, and how to use your portfolio to continue learning and growing in the field of data science.

Uncle Data

Domain Driven Design (DDD) and Infrastructure from Code (IfC) are two powerful approaches to building software. DDD helps developers create flexible, scalable applications and with IfC they can be seamlessly deployed to the cloud. By combining these two approaches, we can create a layered architecture where IfC is just another layer in a DDD app.
To illustrate how we can achieve this, I’ll show an example of an app I developed using DDD principles. To make it work with IfC, I needed to add a configuration layer and use a special Python syntax for the service layer, which enables the IfC engine to compile it. The other layers don't even know that they're running in the cloud, which makes it easy to maintain the application and add new features. This talk will provide insights into how you can leverage the power of DDD and IfC to create robust, scalable, and flexible software applications, and how to incorporate IfC as another layer in your DDD architecture.

More than 5 million people play Nordcurrent mobile games every month. The specificity of free-to-play games is that less than 10% of players make purchases. It is essential to retain paying players and keep them engaged as long as possible. To do that, we built a purchase prediction model.

We store data and make the most of feature engineering in Clickhouse. Apache Airflow orchestrates pipelines. Usually, we use CatBoost for Machine Learning. Pydantic and ClearML, on top of AWS S3, manage model files, training metrics, and configs. The quality in production is evaluated using dashboards in Apache Superset.

The architecture allows us to build fully reproducible ML pipelines. The learning process can be horizontally scaled to select the optimal hyperparameters. At the inference stage, you do not need to worry that the model was trained in some Jupiter Notebook, and it is unclear what to do if it suddenly breaks in a month.

In the current age of AI, the ability to rapidly develop and deploy applications has become crucial for staying competitive. As the demand for AI-powered solutions continues to grow, it's more important than ever to be able to bring new ideas to market quickly and efficiently. The H2O Wave framework is a powerful tool that enables DS/ML people with the required business knowledge to do just that, without the unnecessary overhead of having a software engineering team in the middle.

This talk will introduce H2O Wave, a Python framework that allows developers to build web applications with minimal web development knowledge. With its high-quality UI widgets, built-in authentication, and developer tooling such as IDE extensions, H2O Wave simplifies the app development process and helps teams bring their AI-powered applications to market faster. Attendees will learn how the framework is already being used by Kaggle Grandmasters to build AI applications and how it can help their own development efforts.

pandas 2.0 has recently been released, and one of the key features is a greater support of the Apache Arrow in-memory format. While the change is somehow internal, it opens a wide range of possibilities. In this talk we will have a quick overview of pandas and Apache Arrow, what is new in pandas 2.0, how users will be able to benefit from using pandas with Apache Arrow and what to expect from future pandas releases.

Uncle Data session 2

The word ChatGPT has captured the imagination and the internet, but does ChatGPT truly know everything, is it truly AGI? To answer some of these questions and to formulate what testing a conversational agent trained on a large language model would look like, we had chatGPT take the wit test where we asked it riddles. Find out what it said.
Quick introduction to Large Language Models, how they are trained and what can be improved.
Slides and examples of the test associated with ChatGPT.
Discussion of why ChatGPT fails with understanding contexts, doesn't do well at verbal math, doesn't know what a venn diagram is and has never heard an egg crack and what it means for the next generation of a conversational AI model.
We will go deeper into how it perceives metaphorical language and convoluted relationships, we will also explore how the performance compares to that in the audience to have a good sense of what AGI means.

From recommendation systems to LLM-based applications, vector search is a critical component of the modern AI workflow. Existing vector solutions are complicated to use, hard to maintain, and cost too much. LanceDB is a free open-source vector store that can perform low latency vector search on billion-scale vector datasets on a single node. LanceDB is powered by Lance format, a modern columnar data format for machine learning and data science. Compatible with pandas/polars/duckdb, Lance format supports vector index, predicate pushdown, and random access performance 2000x faster than parquet.

Streamlit is a popular framework for interactive web-based data apps in Python. However, there are some cases where users want to run their apps offline or without sending sensitive data to remote servers. To address these concerns, we introduce 'stlite': a WebAssembly port of Streamlit. It provides offline capability, data privacy, scalability, and multi-platform portability including desktop app packaging, while preserving Streamlit's original features, such as Python productivity and its rich ecosystem.

after a short intro of Streamlit, we will review stlite in the context of the recent emergence of various Wasm-based Python frameworks such as PyScript, and show you what's possible with stlite.
We will also look at its internals from a technical point of view, which may inspire you with ideas on how to make use of Pyodide and how to transform Python frameworks for the Pyodide/Wasm runtime.

You can try out stlite online: https://edit.share.stlite.net/

Uncle Data session 3

Mozilla has been promoting WASM for years, on the other hand, HTMX is gaining attraction. Question is, do we want more frontend or more backend? Do we still need to write JavaScripts?

Building Streamlit apps that enable business users to explore data on their own is an excellent way to support data-driven decision-making in any organization. Pair this up with the animated transitions between the charts provided by the new, open-source library, ipyvizzu, and you have a self-service, interactive report or dashboard that makes it much easier for non-experts to make sense of complex data sets.

Apple’s blocking of the IDFA identifier has made it difficult to attribute Apple users to marketing channels. This poses a big problem to many as it is harder to trace which channels are most effective in driving user-growth. It marks the first step towards the industry having to adapt to a more privacy-centric world where it is harder to track user-level data.

In this talk, Avision will discuss how Mettle have reduced their reliance on user-level data by building a channel-level custom attribution model. This model enabled us to drive efficiencies in re-directing our spend on our strongest channels, leading to higher acquisition at lower cost.

Some of the things we will deep-dive into is why this we use statsmodels instead of scikit-learn; how we benchmark our model’s appropriateness in the absence of an actual target; quickly servicing the insights to drive business-decisions as fast as possible; and then putting it into production via an Apache Airflow and BigQuery.

Have you ever wanted to share Jupyter Notebook with non-technical users? Mercury is a new way to add widgets to a notebook and share it with non-programmers. You can easily build a dashboard, reports, web app, interactive slides, or REST API. Mercury allows you to add widgets to Jupyter Notebook. After the widget change, all cells below the widget are reexetuted with a new widget value. This simple execution model allows converting any Jupyter Notebook into an interactive web application. You can easily create a dashboard or presentation (slides in presentations can be recomputed during the show with values provided with widgets). What is more, Mercury allows schedule automatic execution easily. The framework has a built-in authentication module so that notebooks can be shared publicly or restricted with a password. Mercury is an open-source framework.

DataFrame abstractions are one of the favorite data structures of many data scientists, data-engineers and programmers in general. They offer flexibility and intuitive reasoning on top of query processing.

However, the implementation of DataFrame abstractions have been lacking. On the single node they have been ignoring most research available in RDBMS research. Different from RDBMS, the most known python implementations don't control their own query engines, and are therefore always compromising control, performance and memory usage.

Polars is a DataFrame library that brings a very fast OLAP query engine to the DataFrame abstraction.

This talk we look at what polars has achieved since it's inception and what the future will hold in store.

a

Lightning talks

In this talk we will examine how Python programmers can be creative in the digital age. Whether it's in web development, data science or machine learning creativity is a skill that sets a good developer apart from a great one. Python as a language has been around for 32 years and it may seem like the old saying 'there is nothing new under the sun' is true since so much has been built with it. For this talk we'll look at why building off of the work of other people is essential for creativity and how open source is the thread that ties it all together.

Every business is unique, as are its data and problems. In order to make the most of our valuable data, we need to incorporate these intricacies when building a solution. Bayesian modeling provides a powerful framework for building a so-called digital twin that maps our real-world problem and domain-expertise into a statistical model that can then be fit to data. The benefits are that our solutions are transparent and interpretable by stakeholders, and come with uncertainty measures.

In this talk I will give a few examples of real-world business problems and how Bayesian modeling can solve them. In particular, some common patterns observed in business data sets are time-series, hierarchical or nested structure, and spatial data.

In this talk I will give a few examples of real-world business problems and how Bayesian modeling can solve them. In particular, some common patterns observed in business data sets are time-series, hierarchical or nested structure, and spatial data.

In this talk, we will be exploring what memory profiling is, and how it can help with data science work. We will start the talk with a basic explanation of how Python arrange memories for various objects. This lays the foundation explanation of why we need a special tool to memory profile Python programs.

Then we will be going through a data science use case where we memory profiles some part of the process with the Memray Jupyter plug-in. This would be a use case that a data science practitioner or learner would be familiar with and they can see how memory profiling could be useful.

We will then explain how to interpret the frame diagram in Memray, a commonly used diagram in memory profiling to understand how much memory a process and its sub-process uses. This is something that for a new user, it could be hard to understand and not know what to look into. From this example, audiences can see what they can learn about from the frame diagram.

MLOPs are a collection of practices that enable companies to build, train, deploy, scale, and operate models in production. Model serving is one of the main MLOps tasks. There are multiple ways of running models in production these days: from open-source solutions to enterprise offerings. However, a custom-built serving solution in Python is the most flexible option and can evolve together with your company's needs.

The quality of an ML service is defined by its speed, accuracy, and ability to deal with the load. FastAPI is faster than its predecessors. Also, being a part of the Python ecosystem, it supports all the main ML frameworks. Moreover, it supports the OpenAPI standard out of the box and makes data validation much easier. All of this and its concurrency capability make it a great choice for running ML models in production.

Are you curious about how recommendation engines work? In this workshop, we'll dive deep into the world of TensorFlow Recommender Systems, exploring the fundamental concepts, techniques, and tools needed to build effective recommendation engines. We'll start with an overview of the different types of recommender systems, including collaborative filtering, content-based filtering, and hybrid models. We'll also explore evaluation metrics and learn how to measure the effectiveness of these models.
The workshop then shifts to hands-on exercises that allow you to build your own recommendation engine using TensorFlow. You'll learn how to prepare data, train the model, and make recommendations. Through guided examples, you'll gain a practical understanding of the end-to-end process of building a recommendation engine.

In the usual scenario, Django is served using WSGI server (gunicorn, uWSGI are the most popular) that work in pre-fork synchronous model,
hence you will need multiple workers and have a “copy” of your application for each worker.

This consumes a lot of memory and drags scalability issues along with it,
since one worker is busy responding to only one request at a time.

Green threads are one solution to NOT rewrite your whole codebase in order to go async without bumping into threads and the notorious GIL.

For the demo a simple Django concurrent application will be used and some benchmarks from a load test.

Would you like to contribute to open source projects and you don't know where to start? In this talk we will show you how. It is easier than it sounds when you know the basics on how GitHub is used for open source projects.

In a multi-region company it is not uncommon to encounter difficulties managing users, especially if they use a few products.
I will share the process of creating a platform, including the challenges we faced and lessons learned from building it twice until we finally accomplished our goals.
How we used IDP to enforce security settings, how we migrated the users and how we used it to create new revenue streams.

The talk will introduce Apache Airflow and its competitors. The main goal of the talk is to showcase how Airflow adapted to the ever-changing data space. Comparison and feature exploration would be oriented in tackling the simplest data extraction and transformation layers in different tools. The plan is to showcase extracting part from Database to S3, move to a database, and add a transformation layer to fill the data warehouse and run data quality checks. The comparison would be on speed, efficiency, integration of different tools and vendors (AWS, dbt Labs, Postgres db) and how it looks in the modern data engineering world (adjustments to more frequent refreshes, dataset awareness, community support), what's crucial, what's missing and how it might go in the future for all of these tools.

Leader talks

I'm Customer Success Engineer at JetBrains. Earlier, I led the development of Scala and Kotlin plugins for IntelliJ IDEA. But now, I'm covering the whole scale of our products, including PyCharm.

And here at PyCon LT as I'm ready to listen and discuss your current PyCharm problems. We will exchange for top-level problems and then go deeper into details if have time. Of course, it would be great if you share what you like (new UI? :) ).

We improved the performance, reduced costs, and increased user engagement for the largest B2B pharmaceutical marketplace. Using Python Django, we made the website 4x faster, reduced infrastructure costs by 50%, and decreased bugs by 80%. We also enabled a full CI/CD process, implemented platform alerting and monitoring, and increased delivery rates by 2x. Our team's experience in software development allowed us to revolutionize the marketplace's web development experience, resulting in a 38% growth in organic traffic and a significant increase in website traffic, conversions, and revenue. We also started with a platform built on Next.js that had a performance score of 25, but ended up with a platform that achieved a perfect 100 score on Google. Average platform operation times reduced from 1.2s to 250 ms, while operating on avg 3m requests month.

MLOps tools today are dime a dozen, but do you really need everything to build your machine learning pipelines? If you are just getting started you do not need an army of tools to set up your ML pipelines. In this talk, I will introduce you to the general concept of MLOps, why it is becoming more important these days and then focus on a super interesting MLOps framework in Python called ZenML. ZenML helps you structure your code and pipelines systematically right from the word go, ensuring that you are always building pipelines that can be easily deployed in production. ZenML has a lot of custom components that can be used in different ways. I will take you through the many concepts (steps, pipelines, stacks, integrations) used by ZenML and how you could use them to build your production ready Machine Learning pipelines.

There are 2 most commonly used job schedulers in the Python world: Celery and Dramatiq. Neither of them supports use of Asyncio natively, which can significantly leverage performance of your application.

In this talk, we’ll discuss how you can use Repid to process large quantities of I/O bound tasks. We’ll then dive into the most useful features of the library that will provide you with the perfect framework to get the job done.

After this talk, you will be inspired to unleash the power of Asyncio in your message-driven systems!

AI technologies brough the ability to construct the data in new light. Search engines learn to search not by keywords but by semantic information, the same goes for image or audio search. In this talk we dive into idea and realization how to construct Image/Text/Audio search from scratch, using existing AI pipelines.

Are you struggling with big data in your business? Join us to discover how PySpark can help you solve your problems efficiently and effectively. In this workshop, we will revisit the key concepts of PySpark, including parallel processing and lazy evaluation. We will explore DataFrames as a convenient layer of so called RDDs and work with an optimizer to get the most out of our transformations.

We'll also take a look the Spark UI, which allows us to monitor and optimize our processes. To put our knowledge into practice, we'll simulate a business problem and walk through the entire process of data preparation (preprocessing), training a model with MLLib, and performing inference on preprocessed test data. We'll also add a business logic layer to our solution for further customization (postprocessing).

Optional content includes lessons learned from large-scale production systems based on PySpark. We'll share insights on how to optimize performance and scale your solution to handle big data with ease.

In this talk, we will explore the rapidly evolving field of MLOps. I will delve into best practices and tools that are essential for building and deploying machine learning models. I will cover topics such as data management, hyperparameter tuning, model training and model deployment, also share the latest techniques and tools for streamlining these processes and discuss best practices for monitoring and maintaining machine learning models in production. Whether you are a seasoned machine learning practitioner or just starting out, this talk will provide insights and practical tips for building robust, scalable, and maintainable machine learning systems. Join me as we dive into the world of MLOps and explore the tools and techniques that every machine learning engineer should know.

Python web frameworks such as FastAPI, Flask, Quartz, Tornado, and Twisted are important for developing high-performance web applications and for their contributions to the web ecosystem. However, they may also present certain bottlenecks, either due to their synchronous nature or due to the usage of the Python runtime. These limitations can often arise from their reliance on *SGIs, which can limit the speed and performance of the web application. This is where Robyn comes in. Robyn aims to achieve near-native Rust throughput while still allowing developers to write code in Python. This means that developers can enjoy the benefits of high-performance web applications while still using the language they are comfortable with. In this talk, we will delve into Robyn and explore how it can improve web application performance. We will also discuss the development of Robyn and examine the evolution of a project from ideation to community support. We will examine the challenges and opportunities that arose during the development and how they were addressed.

Most data transformations are written twice. In the field of feature engineering for Machine Learning, data scientists regularly have to build, manage, and iterate on batch jobs, then translate those jobs to a service setting to load data and make fresh predictions. At best, this process is an engineering headache. At worst, this can result in difficult-to-detect deltas between training and inference, complex code, and highly bespoke infrastructure. In this talk we discuss Hamilton, a lightweight open-source framework in python that enables data practitioners to cleanly and portably define dataflows. Hamilton places no restrictions on the nature of transformations, allowing data scientists to use their favorite python libraries. With Hamilton, you can run the same code in your airflow DAG for training as you would in your fastAPI service for inference, and get the same result.

The expectations on developers as well as their self-image has changed many times since the dawn of IT. Their role within an organization today is very different from the past - and more changes are on the horizon.
This talk will provide an overview based on experience with small to large enterprises over the last 30 years. What skills should a developer prepare today, in order to contribute to an organization in the future? What will development work look like in the future?

Domain Driven Design (DDD) is a design approach that puts the business model as the core ground modeling the system’s design and closes the gap between the business logic and code. Amazon Web Services (AWS) empowers DDD with tons of services that can boost your architecture with on-demand databases, message buses, and cloud computing units. And everything connects with a cherry on top - Python’s microservices on serverless AWS resources. In this talk, I will present a use case of how we augmented the client’s team with our experts and helped build the cloud platform for smart metering of IoT devices. The main focus will be put not on theory but rather on showing the technical details and feedback on both: AWS and Python and how they work together to make thousands of devices' data ingestion and data analysis possible for smart buildings.
The outline:
- Short intro about the project
- DDD and how it looks in practice
- Python’s role in this project and tactical patterns when building microservices
- Conclusions what really worked out and what could be avoided
- Q&A

Company presentations 2

For 28 years, the "SciPyData" community has been building foundational software that has influenced science, and engineering and jumpstarted the recent explosion of interest in AI and Machine Learning. in this talk I will briefly review key milestones in the creation of SciPy and PyData communities and discuss key future opportunities and advances in the foundational software behind AI and ML that could effect the next 5 to 10 years.