Bas Harenslak
.ical

Data engineer at GoDataDriven, Airflow trainer and committer, and co-author of the (currently in progress) Manning book Data Pipelines with Apache Airflow.

In his daily job he helps companies become more data driven by building data solutions, and wants to combine cool data products with scalable and solid software. In the past years he worked at various companies such as Booking, ING and Unilever.

The speaker’s profile picture

Sessions

07-15
10:00
45min
Testing Airflow workflows - ensuring your DAGs work before going into production
Bas Harenslak

How do you ensure your workflows work before deploying to production? In this talk I'll go over various ways to assure your code works as intended - both on a task and a DAG level. In this talk I will cover:

  • How to test and debug tasks locally
  • How to test with and without task instance context
  • How to test against external systems, e.g. how to test a PostgresOperator?
  • How to test the integration of multiple tasks to ensure they work nicely together
Amsterdam Meetup, [Sessions start: Wednesday 15.07 6pm (Wednesday 15.07 9 am PDT)]