07-10, 10:00–10:45 (US/Pacific), Warsaw Meetup [Session starts: Friday 10.07 6pm (Friday 10.07 9am PDT)]
Airflow does not currently have an explicit way to declare messages passed between tasks in a DAG. XCom are available but are hidden in execution functions inside the operator. AIP-31 proposes a way to make this message passing explicit in the DAG file and make it easier to reason about your DAG behaviour.
In this talk, we will explore what other DSL are doing for message passing and how has that influenced AIP-31. We will explore the motivations behind explicit message passing as well as more proposals that can be built on top of it. In addition, we will explore a new way to define custom Python transformations using the task decorator proposed, and how this change may improve the extensibility of Airflow for more experimental ETL use cases.
Software engineer with special interest in machine learning and data engineering. Graduated with a MS in Computer Science on September 2019 from University of Colorado Boulder. For his master thesis he designed and implemented an ingestion system for social media messages produced during disasters. This work currently helps analysts at the EPIC laboratory to analyze and support the information needs by members of the public during times of mass emergency. In his free time, he likes to ski, swim, take pictures and read science fiction novels.