2025-12-08 –, Thomas Paul
In this hands-on tutorial, you'll go from a blank notebook to a fully orchestrated data pipeline built entirely in Python, all in under 90 minutes. You'll learn how to design and deploy end-to-end data pipelines using familiar notebook environments, using Python for your data loading, data transformations, and insights delivery.
We'll dive into the Ingestion-Tranformation-Delivery (ITD) framework for building data pipelines: ingest raw data from cloud object storage, transform the data using Python DataFrames, and deliver insights via a Streamlit application.
Basic familiarity with Python (and/or SQL) is helpful, but not required. By the end of the session, you'll understand practical data engineering patterns and leave with reusable code templates to help you build, orchestrate, and deploy data pipelines from notebook environments.
Modern notebooks have evolved far beyond exploratory data analysis. They can now serve as complete, reproducible environments for building and orchestrating data pipelines. In this 90-minute tutorial, participants will learn how to leverage a notebook as data engineering workspace, capable of deploying an end-to-end data pipeline that can ingest and transform data, and deliver insights to end users.
I’ll introduce the Ingestion–Transformation–Delivery (ITD) framework as a practical model for structuring data workflows, while learners follow along with hands-on coding exercises. Attendees will build and automate their own pipeline, using Python libraries to load, transform, and visualize data.
Topics covered:
- Understanding the ITD framework for data engineering
- Data ingestion from CSVs and cloud object storage
- Transformations using Python and DataFrame APIs
- Delivery patterns: writing to databases/tables and building Python data applications
- Lightweight orchestration and task scheduling, directly in notebooks
Key takeaways for learners:
- Learn a reproducible framework for notebook-driven data pipelines
- Use Python for your end-to-end workflow (and combine it with SQL as-needed)
- Automate, and schedule pipelines entirely from within a notebook
By the end of the session, participants will have a working pipeline and reusable, boilerplate code templates for building and deploying data pipelines.
Gilberto has spent over a decade shaping technical developer education worldwide. To date, he's made complex concepts accessible to over 100,000 students and engineers through both online learning platforms and in-person experiences.
At Codecademy, he authored and launched several of their foundational courses. Since then, he's worn multiple hats as both product manager and technical content creator at industry leading companies, including MongoDB, Domino Data Lab, Plaid, and Snowflake.
Gilberto is passionate about crafting exceptional developer experiences and educational resources. He frequently writes about data engineering, AI, and application development.
Connect with him on LinkedIn: https://www.linkedin.com/in/gilberto-hernandez/