07-07, 21:00–21:45 (US/Pacific), Bangalore Meetup [Session starts Wednesday 8.07 9.30 am (Tuesday 07.07. 9pm PDT) ]
With data becoming the new oil, Data Reliability Engineering(DRE) has been buzzing all around in all leading Fin Tech industries. And in this space, working on big data/data science necessarily deals with building pipelines starting from data exploration to enrichment at scheduled intervals. While simple scripts are easy to create it gets cumbersome to manage when we want to build a resilient data pipe with intelligent failure handling for unexpected long running/short lived dependent tasks. Apa
Ever wondered how could we play Spotify within Uber App or view the destination location via Google maps in your bookstore’s website?
It’s the API’s that enable distant and different entities to communicate with each other in a standard format and nevertheless we do have Airflow API’s too that are in its initial stages for consumption. And in PayPal we wanted to move away from two of our enterprise schedulers Control-M and UC4 to Airflow as we faced several challenges with respect to onboarding and management in the existing scheduling platform.
And as we started the journey, the first most important step that we wanted to take was to build all the mandatory API’s on the top of Airflow so that we can integrate with our Self-Service Tools.
//Continued in the notes section
Aishwarya Sankaravadivel is a senior technologist leading the Scheduling Platform at PayPal having deep expertise in managing Airflow & Elasticsearch for global customers. She is a passionate software engineer having intense hands-on coding experience in Scala/Python and Reactive Programming. She has grabbed several awards like CPI India Champion, Wonder Woman, Star of the month for her exemplary work at PayPal. She has spoken at several conferences in India and Asia Pacific regions and also a Women in Data Science Ambassador who has organized multiple meetups in India. Besides work, she is an avid reader and reviewer of many books and novels.