07-13, 10:00–10:45 (US/Pacific), London Meetup, [Sessions start: Monday 13.07 5pm (Monday 13.07 9am PDT)]
Learn how Devoted Health went from cron jobs to to Airflow deployment Kubernetes using a combination of open source and internal tooling.
Devoted Health, a Medicare Advantage startup, went from cron jobs to Airflow on Kubernetes in a short period of time. This journey is a common one, but still has a steep learning curve for new Airflow users. This talk will give you a blueprint to follow by covering the tools we use, best practices, and lessons learned. We'll share Devoted's approach to managing our deployment, monitoring the platform, and developing, testing, and deploying DAGs. This includes internal tooling we've written that allows Data Scientists to work with Airflow without worrying about Airflow itself.
Adam Boscarino is a Data Engineer at Devoted Health working to improve healthcare in America. Previously, he worked at DigitalOcean with open-source data tools like Apache Spark, Kafka, and Airflow. And before that, he worked at Fitbit making sure all your steps were properly counted using tools like Luigi, Looker, and Snowflake.