PyData Amsterdam 2026

A typical data platform often assumes a reliance on scalable computing solutions, ensuring that the same data processing jobs can remain efficient and cost-effective while the amount of data grows. However, in reality, this assumption does not always hold true when meeting the evolving demands of the business.

Data engineering practitioners face a critical dilemma: hardware costs grow proportionally as the business scales, while switching to a more performant solution implies prohibitive engineering costs due to the need to rewrite entire workloads.

This pattern was observed a decade ago when teams moved from MapReduce systems to then-innovative solutions such as Apache Spark. The same theme occurs today, as Spark is challenged by successor compute systems designed specifically for modern cloud environments.
This talk discusses the modernization of Apache Spark workloads. The goal is to achieve orders-of-magnitude improvements in compute time and hardware cost without requiring code changes. The solutions largely fall into two categories:

1. Spark accelerators. These solutions replace physical operators with highly optimized ones written in other programming languages. The optimized operators are injected using the Spark plugin mechanism.
2. Alternative server implementations of the Spark Connect protocol. These solutions allow Spark clients to connect to them via gRPC for the same DataFrame operations. Such solutions present an opportunity to eliminate the deficiencies of JVM-based data systems entirely.

The talk will discuss the pros and cons of both categories of solutions and present an empirical study of their performance. It will also explain the common technologies that make these performance improvements possible: the Apache Arrow in-memory format and the Rust programming language.

As a concluding case study, this talk will discuss how custom Python logic (UDFs and data sources) is supported in Spark, and how modernization efforts can be driven by Rust and PyO3. Python code is historically known to be slow in Spark due to the friction between the Python runtime and the JVM. However, this modernization effort can make Python workloads highly performant in Spark. This opens up a wide range of possibilities in agentic workflows, where standard data processing is often blended with custom business logic and integrations.

This talk is intended for data scientists, data engineers, and ML engineers interested in scalable ETL pipelines and OLAP workloads. The audience will not only be equipped with a handbook for the production-ready modernization of Spark workloads, but will also gain an in-depth understanding of the current trends in modern data system innovations.

For the audience, prior experience with Apache Spark is nice to have but not required. Knowledge with data system internals is not needed as this talk serves for educational purposes on such topics.

Here is an outline of the talk:

• Problem description (2 minutes)
• Overview of Spark workload modernization solutions (5 minutes)
• Techniques behind the scenes (8 minutes)
  • Arrow in-memory format
  • Query planning and execution in Rust
• Performance comparison and discussions (5 minutes)
• Case study: Performant Python in Spark (5 minutes)
• Q & A (5 minutes)