2020-07-08 –, Bay Area Meetup [Session starts: Wednesday 08.07 9 am (Wednesday 08.07 9am PDT)]
To share the experience of adopting Airflow as the next generation of workflow system at pinterest, distributing the worker loads k8s, load sharding/partitioning via multiple schedulers, transparent existing user flow migration and more .
The current work flow system , pinball, has served the data pipeline orchestration demands here at Pinterest well for years, however with the rapid increasing execution demand the system started to expose scalability and performance issues. As the fix address the issue in pinball might be intense the workflow team at Pinterest decided to look for a new solution to better address the issues and serve the workflow scheduling demand, and we chose Airflow as our next generation of workflow.
In this talk we want to discuss how we made the decision to on board to airflow, and beyond the out-of-box features and experience what improvements we made to better support the business need at Pinterest.
Software Engineering at Pinterest. Working on workflow systems, including Airflow.
I'm a Software Engineer @ Pinterest Workflow Team in the Data Org. I have been working on Airflow for almost a year and the team is in the progress of migrating from the old workflow platform to Airflow. Before joining Pinterest, I received my master degree from Carnegie Mellon University.
Hi all,
I'm currently a software engineer on the big data platform team in Pinterest where our focus is on the workflow platform. We are migrating from old systems to this new one, which is built around airflow. Prior to here, I was on the data engineering team at Pandora where I first started exploring workflow systems and brought in a service built around Airflow there.
Outside of software, I help my family operate two pizza shops so coding and cooking are in my DNA!