Lucas Mendes Mota da Fonseca
I am a Data Engineer at QuintoAndar.
I firstly graduated in System Analysis and Development and then studied Big Data and Data Management Intelligence at Polytechnic School of the University of Sao Paulo. I started working as a Data Engineer at QuintoAndar, where I knew Apache Airflow and its vibrant community.
Our data team daily works towards making our data platform better to continue supporting QuintoAndar as a data-driven company.
I am passionate about learning and sharing ideas, so I will be glad to talk about data architecture or any Airflow related topics.
Data Engineer at QuintoAndar
Session
Cross-DAG dependency may reduce cohesion in data pipelines and, without having an explicit solution in Airflow or in a third-party plugin, those pipelines tend to become complex to handle. That is the reason we, at QuintoAndar, have created an intermediate DAG to handle relationships across data pipelines called Mediator, in order for them to be scalable and maintainable by any team.