The current zeitgeist is that the data lake concept from classical data engineering and modern data warehousing from business intelligence are converging more and more. This is also driving the shift from ETL to ELT, and so tools such as dbt are becoming increasingly important in combination with modern Big Data warehouses such as Snowflake and Google BigQuery. For typical data and MI engineers, this is quite a departure from familiar tools like Spark.

Having a pure Spark and ETL background myself, this trend motivated me to explore the foreign realms of ELT, data warehousing and especially the fuzz about dbt. In this talk I want to share my key insights with classical data / ml engineers that might have only heard about Snowflake, dbt, Airbyte and Lightdash but have never cared to dig deeper.

My talk is structured like this:
* short introduction to the differences of data lake vs data warehouse, ETL vs ELT
* high-level introduction of Snowflake, Airbyte, dbt, and Lightdash
* demonstration based on the Kaggle Formula 1 World Championship dataset to see those four tools in action
* my main take-aways and key insights

After this talk, you will have learned the differences between ETL & ELT, what these four tools do and in which cases you should consider the WALD stack. Also, you will know how to use Python instead of SQL to define models in dbt, which is a brand-new feature.

The WALD-stack is sustainable since it consists mainly of open-source technologies, however all technologies are also offered as managed cloud services. The data warehouse itself, i.e. Snowflake or Google BigQuery, is the only non-open-source
technology in the WALD-stack. In my talk, I will focus on the open-source parts of the WALD-stack.