2026-09-11 –, Room 1 (170)
For data engineers and data scientists building Python pipelines workloads can hit the awkward middle: too large to handle comfortably with pandas on a single machine, too small for a Spark cluster to justify the operational cost. The usual choice is overprovisioned single machines on one end, or accept cluster maintenance and a second DataFrame API.
This talk argues that you don't need to choose. Polars' new streaming engine is the common core behind high performance single node processing with spill-to-disk capabilities, and distributed execution. The same API scales with your data.
Attendees will leave with actionable insights that help decide: when vertical scaling and spill-to-disk are enough (which gets you further than most assume), when distributed computing is actually justified, and technical knowledge of how Polars accomplishes both.
We'll follow one query across different dataset sizes, go through the technical challenges we overcame, and teach you to decide your optimal choice for each scenario.
Audience and prerequisites: Attendees should be comfortable with a DataFrame API. Prior Polars experience is helpful but not required. No distributed-systems background assumed. Concepts like "shuffles" and "partitioning" are introduced as they come up.
Structure: A presentation with slides that follow one query through different dataset sizes:
- 10 GB on a single-node: morsel-driven, streaming execution; how we optimize performance on the single node.
- 100 GB, single-node: how a single node can process more data than expected.
- 1 TB, vertical scaling vs horizontal scaling: how out-of-core computation stretches a single node to its maximum and when the jump to distributed execution is worth the overhead.
- 10TB distributed: three technical challenges a distributed DataFrame engine has to solve: Plan partitioning at stage boundaries, data shuffling between nodes (and how to minimize it), and worker coordination.
The talk pairs engine internals with visualisations that support decision making based on scenarios.
Thijs Nieuwdorp is the DevRel Engineer at Polars in Amsterdam. His interest in the interaction between human and computer led him to an education in Artificial Intelligence at the Radboud University, after which he dove straight into the field of Data Science. At Xomnia he witnessed the birth of Polars as Ritchie Vink started working on it during his employment there , and has been using it in his projects ever since. Outside work Thijs enjoys exploring our world through hiking and traveling, and exploring other worlds through books, games, and movies.