2021-10-21 –, Workshops
Live Stream: https://youtu.be/qWvJSIgOcPU
With a lot of changes under the hood with Airflow 2.0, the workshop aims to give an overview on major updates in Airflow 2.0 from 1.0, major components and working of Airflow and hands-on demo of implementation and management of an end-to-end Machine Learning pipeline. Without a pipeline in-place, management of multiple Machine Learning stages in production can be difficult. This gives an overview of simplified process and management of Python based ML projects using Airflow.
Prerequisites
- Install Docker Desktop (with minimum 3GB memory allocated)
- Start Docker engine
- Clone the workshop repo with
git clone https://github.com/pycon-ml/airflow_workshop.git - Run
docker-compose pullinside repo folderairflow_workshop
Agenda
05 min: Introduction
05 min: Major changes in Airflow 2.0
05 min: Pre-requisites setup overview
10 min: Walkthrough of different backend components
10 min: Different stages of a DAG file – steps and operators
10 min: Dynamic DAG creation to improve parallelism
15 min: How to trigger Airflow DAG runs
15 min: Debug and clear Airflow task errors
10 min: Overview of production-level Airflow-based architecture
05 min: Wrap up questions
Alen Jacob is a Machine Learning Engineer at H&M and have a Masters' Degree in Computational Linguistics.
Scott Zhou is a competence lead for Machine Learning Engineers at H&M and a Machine Learning Engineer himself.
Lini Jose is a Machine Learning Engineer at H&M.
Nitin Bisht is a Software Engineer at H&M.