2020-07-10 –, Warsaw Meetup [Session starts: Friday 10.07 6pm (Friday 10.07 9am PDT)]
One of the significant challenges in scaling Airflow at an organization is the number of qualified developers fluent in Python. To speed the development of complex pipelines we developed a DAG authoring and editing tool for Airflow. Installed as a plugin, this tool allows users to author DAGs compose existing operators and hooks with virtually no Python experience.
A live demo of the tool and accompanying code.
This project allows Data Engineers and other staff to quickly create dags out of existing operators, hooks, SubDags, and sensors. By moving the creation of the DAG out of the text editor, DAGs can be quickly rearranged, edited, and tested. In addition, Data Scientists, Analysts, and other users can create automation and ETL pipelines without requiring extensive Python knowledge.
We will walk you through a live demo of DAG authorship and deployment, and spend time reviewing the underlying open-source standards used and the general approach that was taken to develop the code.
In addition to allowing dags to be created in a visual editor, the underlying tech enables Airflow DAGs to be described programmatically in YAML or JSON. DAGs described there can be saved in backing databases instead of python files.
Placeholder biography