2024-09-25 –, Gaston Berger
Polars is a dataframe library taking the world by storm. It is very runtime and memory efficient and comes with a clean and expressive API. Sometimes, however, the built-in API isn't enough. And that's where its killer feature comes in: plugins. You can extend Polars, and solve practically any problem.
No prior Rust experience required, intermediate Python and programming experience required. By the end of the talk, you will know how to write your own Polars Plugin! This talk is aimed at data practitioners.
Have you ever had the experience of needing to write a really custom function? Did you end up using a custom Python lambda function and waiting endlessly whilst your code executed?
Let's learn how to put an end to that!
This talk is aimed at advanced dataframe users who want to go beyond what Polars offers them. The structure will be:
- 5 minutes motivation: example of a custom function which is painfully slow
- 5 minutes prerequisites: what's Polars Series? What's a Chunked Array? What's an Arrow Array? What's a validity mask?
- 5 minutes Rust basics: the bare minimum, and recommended resources to learn it properly
- 5 minutes: how to translate Python lambda functions to Rust plugins: elementwise single-column example (take the absolute value of each element), then non-elementwise example (cumulative sum)
- 5 minutes addressing the motivating example with a Polars plugin, show how much faster it became with only ~30 lines of Rust code
- 5 minutes: Q&A
By the end of the session, attendees will know how to write their own Polars Plugin, and how to go about learning any prerequisites they may currently lack (such as Rust). This talk is aimed at data practitioners who have experience with Python and data analysis (however, no prior Rust experience is required!).
Marco is a core dev of pandas and Polars and works at Quansight Labs as Senior Software Engineer. He also consults and trains clients professionally on Polars. He has also written the first Polars Plugins Tutorial and has taught Polars Plugins to clients.
He has a background in Mathematics and holds an MSc from the University of Oxford, and was one of the prize winners in the M6 Forecasting Competition (2nd place overall Q1).