Aleksandr Eliseev
Work in Data Engineering at Wrike since August 2016.
Migrated product data engineering ETLs between Spark clusters
Leading the migration from our own hardware to GCP and BigQuery
Make data available to engineers across the company
Session
07-14
09:00
45min
How do we reason about the reliability of our data pipeline in Wrike?
Aleksandr Eliseev
We’re using airflow for almost two years now and scaled it from two users to 8 teams.
We would like to share our story, how we reason about the reliability of our data pipelines.
We will tell:
How are we establishing a reliable review process on AirFlow?
How we’re using multiple-airflow configuration to migrate from our DC to cloud and to reuse the production data in acceptance wherever possible.
How do we use data versioning to make sure that data is up-to-date throughout the pipeline?
Deployments
NYC Meetup,[Sessions start: Tuesday 14.07 12pm (Tuesday 14.07 9am PDT)]