Reuniting the two distant cousins: Orchestrating your end-to-end Data Engineering Workflow Leveraging Python in Apache Beam and Apache Airflow
2026-09-12 –, Room A All times in Europe/AmsterdamPart 1: Introduction & The Problem (10% of time): Introduce Apache Beam and Apache Airflow as "distant cousins" in the data ecosystem and highlight the common challenge of bridging the gap between large-scale data processing and workflow orchestration.
Part 2: The Core Concepts (20% of time): Explain the distinct, complementary roles of each tool, defining Apache Beam's strength in unified, large-scale data processing (batch/streaming) and Apache Airflow's role in scheduling, monitoring, and managing complex workflows, both using beginner-friendly Python.
Part 3: The Live Build: An End-to-End Workflow (50% of time): Provide a practical demonstration showing how a Python-based Airflow DAG can trigger a Python Apache Beam pipeline for a data processing task, wait for its completion, and then orchestrate subsequent steps in the workflow.
Part 4: Cloud Integration & Key Takeaways (20% of time): Showcase how to extend the pipeline by integrating with Google Cloud services like triggering a Cloud Function, loading results into BigQuery, sending a completion notification via SendGrid, and making a call to a Gemini AI model for a final, value-add step.
This talk explores the synergy between Apache Beam and Apache Airflow, demonstrating how to create a robust, end-to-end data engineering workflow using Python. We will dive into the challenges of orchestrating complex data processing tasks and show how combining Airflow's scheduling capabilities with Beam's data processing framework can create more efficient and manageable data pipelines. Both Airflow and Beam processes will be written in beginner-level python - so if you are a beginner or intermediate data engineer or architect, this session is for you! Lastly, we will cover integrations with Cloud Platform services, including Cloud Functions, BigQuery, SendGrid email, and the latest Gemini AI models.
Sadeeq is a Data Analytics Specialist at Google Cloud in the UK. His role involves understanding customers' Data Engineering and Analytics challenges and goals, while helping them through their Digital Transformation journeys as they leverage solutions primarily on Google Cloud Platform, as well as on-Prem or on other Clouds. Whilst in Nigeria, Sadeeq worked as a Software Engineer at few Startups, but gradually transitioned into Data Engineering at FMDQ Group. He then moved to Portugal for an MSc. degree in Data Science and Advanced Analytics at NOVA University of Lisbon. He has previously worked at KPMG and Microsoft, and his almost two decades of industry experience include consulting with and for other notable Fortune 500 companies on data-centric projects.