Stefan Dienst
Stefan is a data engineer and works at Covestro in a newly established data office. He has four years of experience working on a variety of data platforms, ranging from classic ETL pipelines and data warehousing to near–real-time stream processing. Before moving into data engineering, he completed a PhD in physics, where he felt in love with Python and working with data. Since then he is always curious to learn new things and share what he has learned with others.
Session
In the daily work of a data engineer, building new data pipelines often takes priority, while maintaining them and ensuring their correctness becomes an afterthought. This focus can quickly turn into a pitfall: failures go undetected, incorrect data silently propagates, and complaints from stakeholders arrive before engineers notice any issues. In practice, incorporating observability into every new data pipeline helps avoid these problems and enables teams to steadily increase system complexity while maintaining trust and peace of mind.
In this talk, I introduce observability in the context of data pipelines, covering its three core pillars: metrics, alarms, and logs. We will explore concepts like the four golden signals, alarm fatigue and structured logging and how they apply to data pipelines. I will show easy to implement first steps and share real-world experiences, where improved observability helped uncover previously unknown incorrect behavior and build trust in data systems.
This talk is well suited for data engineers that had little exposure to observability and want to learn about strategies how to keep sane while managing a jungle of pipelines.