Santosh Pingale
Santosh Pingale loves taming large-scale infrastructure problems with open source. His interests span data platforms, distributed systems, performance engineering, and the resilience challenges that appear once systems meet the real world.
Session
As data volumes grow exponentially, data engineering teams face a critical dilemma: hardware costs steadily escalate, yet migrating to a more performant system often demands a prohibitively expensive rewrite of existing workloads. How can organizations achieve orders-of-magnitude improvements in performance and drastic cloud cost reductions without touching a single line of their legacy Apache Spark code?
This talk explores the cutting-edge landscape of Spark modernization. We will evaluate two primary drop-in replacement strategies: Spark accelerators that inject highly optimized physical operators via plugins, and alternative server implementations that leverage the Spark Connect protocol to bypass JVM inefficiencies entirely. Attendees will look under the hood to see how modern technologies, specifically the Rust programming language and the Apache Arrow in-memory format, power these massive performance leaps.
Furthermore, we will showcase a case study on running Python workloads (UDFs and custom data sources) at unprecedented speeds within the Spark ecosystem, demonstrating how Rust and PyO3 can successfully eliminate the historical friction between Python and the JVM.
This talk is designed for data engineers, ML engineers, and data scientists handling intensive ETL or OLAP workloads. The talk delivers actionable insights to bring modern, high-performance data system innovations into production.