2025-04-24 –, Titanium3
Orchestration is a typical challenge in the data engineering world. Scheduling your data transformation jobs via CRON-jobs is cumbersome and error-prone. Furthermore, with an increasing number of jobs to manage it gets in-oversee able. Tools like Apache Airflow, Dagster, Luigi, and Prefect are known for addressing these challenges but often require additional resources or investment. With the advent of serverless orchestration tools, many of these disadvantages are mitigated, offering a more streamlined and cost-effective solution.
This session provides a comprehensive overview of combining serverless architecture with orchestration. We will start by defining the core concepts of orchestration and serverless technologies and discuss the benefits of integrating them. The talk will then analyze solutions available in the cloud vendor space. Attendees will leave with a well-rounded understanding of the tools and strategies available in serverless orchestration.
Orchestration is a typical challenge in the data engineering world. Scheduling your data transformation jobs via CRON-jobs is cumbersome and error-prone. Furthermore, with an increasing number of jobs to manage it gets in-oversee able. Tools like Apache Airflow, Dagster, Luigi, and Prefect are known for addressing these challenges but often require additional resources or investment. With the advent of serverless orchestration tools, many of these disadvantages are mitigated, offering a more streamlined and cost-effective solution.
Beyond data engineering, serverless orchestration holds substantial potential for classical software engineering, especially as organizations explore serverless approaches for optimizing efficiency and reducing overhead.
In this talk you will explore:
* Basic Introduction to Serverless Orchestration:
- What is orchestration about?
- What is serverless about?
- Why combining the two of them?
* Offerings from Major Cloud Vendors:
- Analyzing solutions from leading cloud providers in the realm of serverless orchestration
* Patterns and Solutions for Serverless Orchestration in Software Engineering:
- Exploring how serverless orchestration can be applied within classical software engineering contexts
Participants will leave this session equipped with a comprehensive understanding of the serverless orchestration landscape and its applications across different engineering disciplines.
Novice
Expected audience expertise: Python:Novice
Tim is a Data Engineer at Cloudflight, based in Innsbruck. There he architects and builds modern data infrastructures in customer projects, ranging from streaming ETL pipelines to data catalogs and entire data platforms. His focus areas include software engineering, cloud technologies, data platform engineering, and DataOps. Tim is also a passionate Open Source contributor, actively working on projects like Apache StreamPipes and DataHub, among others.