We are from the data team of viadukt, a start-up that aims to increase the energy-efficient modernization of buildings across Germany. To achieve this, we set up a comprehensive geospatial database containing building-related data and developed a processing pipeline that integrates both federal and open-source datasets. Initially, PostgreSQL with the PostGIS extension met our needs while we focused on North Rhine-Westphalia. But as our scope expanded nationwide, significant challenges arose.

PostgreSQL required costly vertical scaling, even though we only occasionally needed high processing power, resulting in an underutilized yet expensive setup. Alternatively, we could have downsized our machines whenever processing was not relevant. However, this would require downtime of our instance, which we wanted to avoid, or complex mechanisms to avoid it.

To address these issues, we implemented a solution based on Apache Spark and Apache Sedona that met specific criteria: cost-effectiveness, flexible scaling for storage and compute, consistent performance, and an easy migration path. In this talk, we'll explore how we overcame challenges we faced during the migration and the solutions we implemented.