Thijs Nieuwdorp
Thijs Nieuwdorp is the DevRel Engineer at Polars in Amsterdam. His interest in the interaction between human and computer led him to an education in Artificial Intelligence at the Radboud University, after which he dove straight into the field of Data Science. At Xomnia he witnessed the birth of Polars as Ritchie Vink started working on it during his employment there , and has been using it in his projects ever since. Outside work Thijs enjoys exploring our world through hiking and traveling, and exploring other worlds through books, games, and movies.
Session
For data engineers and data scientists building Python pipelines workloads can hit the awkward middle: too large to handle comfortably with pandas on a single machine, too small for a Spark cluster to justify the operational cost. The usual choice is overprovisioned single machines on one end, or accept cluster maintenance and a second DataFrame API.
This talk argues that you don't need to choose. Polars' new streaming engine is the common core behind high performance single node processing with spill-to-disk capabilities, and distributed execution. The same API scales with your data.
Attendees will leave with actionable insights that help decide: when vertical scaling and spill-to-disk are enough (which gets you further than most assume), when distributed computing is actually justified, and technical knowledge of how Polars accomplishes both.
We'll follow one query across different dataset sizes, go through the technical challenges we overcame, and teach you to decide your optimal choice for each scenario.