Lawrence Mitchell
Lawrence Mitchell works at NVIDIA. His focus is on high-productivity, high-performance libraries for data analytics. He leads the technical design and implementation of the cuDF-accelerated Polars GPU engine. Prior to joining NVIDIA he was a lecturer in computer science and applied mathematics at the University of Durham with research interests in high performance simulation of continuum mechanics, structure-preserving numerical methods, and preconditioning techniques for coupled multiphysics problems. He was a founding co-lead and technical architect of the open source Firedrake project for finite element simulation.
Session
Large scale analysis of structured data underpins much of enterprise decision making. These analyses are often formulated using dataframe libraries, of which Polars is a popular example. It has a domain specific language for writing dataframe queries, exposed as a lazy API in Python, a query optimiser and multiple different execution engines.
In this talk, I will cover recent work developing the multi-GPU accelerated engine in Polars. I will give an overview of the computational patterns that appear in large scale data analytics, where they present challenges to efficient execution and how we address them. I'll show how this multi-engine offering allows seamless scaling of data analyses from laptop, to workstation, and beyond, providing an "interactive" experience even at terabyte scale.
If you've ever written dataframe code and wondered "what is actually going on here?", this talk might be for you.