Aleksandr Eliseev
Work in Data Engineering at Wrike since August 2016.
Migrated product data engineering ETLs between Spark clusters
Leading the migration from our own hardware to GCP and BigQuery
Make data available to engineers across the company
Team Lead of Product Data Engineers @ Wrike
Sessions
We’re using airflow for almost two years now and scaled it from two users to 8 teams.
We would like to share our story, how we reason about the reliability of our data pipelines.
We will tell:
How are we establishing a reliable review process on AirFlow?
How we’re using multiple-airflow configuration to migrate from our DC to cloud and to reuse the production data in acceptance wherever possible.
How do we use data versioning to make sure that data is up-to-date throughout the pipeline?