2026-09-11 –, Unconference
Delta Lake and Apache Iceberg solved data lake chaos but introduced a new one: layers of JSON metadata scattered across object storage. DuckLake is an open table format that challenges the status quo by storing all catalog metadata in a plain SQL database rather than a sprawling file system.
In this talk we will explore DuckLake's architecture, which couples the low-cost storage of S3 with the transactional guarantees of a standard SQL database. Using only Python and DuckDB — no Spark cluster required — we will demonstrate how to build a fully ACID-compliant lakehouse, complete with schema management, time travel, and concurrent writers.
We will also candidly discuss where DuckLake falls short: the single-writer bottleneck, the scale inflection point where a distributed catalog becomes necessary, and the ecosystem maturity trade-offs of adopting a newer format over established giants.
Audience: data engineers, analytics engineers, and data scientists working with lakehouse architectures or curious about lightweight alternatives to Iceberg/Delta.
Takeaways: understand the "metadata-in-database" pattern, learn to deploy a serverless multi-user lakehouse with the Python ecosystem, and gain a pragmatic framework for choosing between lightweight and enterprise-grade approaches.
Motivation:
The modern data lakehouse stack has grown remarkably complex. Formats like Apache Iceberg and Delta Lake brought ACID guarantees to object storage, but at the cost of a metadata layer made of JSON manifests, Avro snapshots, and eventually-consistent file listings. Practitioners now debug metadata issues more than data issues. DuckLake proposes a fundamentally simpler approach: use a regular SQL database (PostgreSQL, MySQL, SQLite, or DuckDB itself) as the single source of truth for all table metadata, while keeping data files in cheap blob storage.
What will be covered:
- The problem with file-based metadata: A brief review of how Iceberg and Delta manage metadata through manifest files and transaction logs, and the operational pain points this creates (manifest compaction, orphan files, catalog server dependencies).
- DuckLake architecture: How metadata is stored in relational tables, how snapshots and schema evolution work, and why this approach provides true multi-table ACID transactions out of the box.
- Live demo: Using Python and DuckDB we will:
- Create a DuckLake catalog backed by a PostgreSQL database and S3-compatible storage
- Build schemas, ingest data, and query tables
- Demonstrate time travel (querying historical snapshots)
- Show concurrent writer behavior and conflict resolution
- Honest limitations: Where DuckLake struggles today:
What will not be covered:
- Spark-based lakehouse deployments, cloud-managed catalog services (AWS Glue, Databricks Unity Catalog), or deep internals of the DuckDB execution engine.
Currently working as a Machine Learning Engineer at Intella.
Interested in end-to-end Machine Learning pipelines, MLOps, Data Engineering and deploying production-ready scalable systems.
Previous experience in the field of Computer Vision where I’ve been able to publish a co-authored paper during graduation. During my academic experience at the University of Bari I’ve worked on the ISO 25010 certified project e-GLU BOX: a platform for usability tests made for PA. The application was developed both as a web application using Laravel and as a mobile app using Flutter.
Technical skills:
- Python, Pandas, Polars, Numpy, Pyspark, Dash, Prefect
- Delta Lake, Iceberg, DuckDB, Databricks, Azure
- Pytorch, Keras, Tensorflow, Sklearn
- MLFlow, SHAP, Docker
- Flutter, Laravel
Domains of interest:
- Artificial Intelligence, Machine Learning, Data Science, Time Series Forecasting, Computer Vision, Data Engineering