DuckLake: The Lakehouse That Finally Embraces the Database
Delta Lake and Apache Iceberg solved data lake chaos but introduced a new one: layers of JSON metadata scattered across object storage. DuckLake is an open table format that challenges the status quo by storing all catalog metadata in a plain SQL database rather than a sprawling file system.
In this talk we will explore DuckLake's architecture, which couples the low-cost storage of S3 with the transactional guarantees of a standard SQL database. Using only Python and DuckDB — no Spark cluster required — we will demonstrate how to build a fully ACID-compliant lakehouse, complete with schema management, time travel, and concurrent writers.
We will also candidly discuss where DuckLake falls short: the single-writer bottleneck, the scale inflection point where a distributed catalog becomes necessary, and the ecosystem maturity trade-offs of adopting a newer format over established giants.
Audience: data engineers, analytics engineers, and data scientists working with lakehouse architectures or curious about lightweight alternatives to Iceberg/Delta.
Takeaways: understand the "metadata-in-database" pattern, learn to deploy a serverless multi-user lakehouse with the Python ecosystem, and gain a pragmatic framework for choosing between lightweight and enterprise-grade approaches.
Modern lakehouse formats brought ACID transactions and schema evolution to object storage, but they also introduced substantial metadata machinery. Apache Iceberg uses JSON metadata and Avro manifests, while Delta Lake maintains a transaction log and periodic checkpoints. These designs are proven and widely supported, but operating them can involve metadata maintenance, log processing, orphan-file cleanup, and additional catalog infrastructure.
DuckLake takes a different approach: it stores lakehouse metadata in ordinary SQL tables while keeping table data as Parquet on local or object storage. Wether it's PostgreSQL, SQLite, or DuckDB, each of them can act as the transactional catalog and single source of truth. With release v1.0 DuckLake is now production-ready, with a stable specification and backward-compatibility guarantees.
What will be covered
Why SQL-backed metadata? A concise comparison with Iceberg manifests and Delta transaction logs, including the operational trade-offs of each approach.
DuckLake architecture: How relational catalog tables represent schemas, snapshots, statistics, partitions, and data files—and how this enables snapshot isolation and multi-table ACID transactions.
Live demo with Python and DuckDB:
Connect to a PostgreSQL-backed DuckLake catalog and inspect its metadata
Ingest and query a managed table
Execute an atomic multi-statement transaction
Query historical snapshots and inspect changes through the change data feed
Demonstrate schema evolution, upserts, and maintenance operations
Limitations: The catalog database becomes critical infrastructure; access control currently relies on the catalog and storage systems; heavily conflicting writes can exhaust retries; and the broader multi-engine ecosystem is still less mature than Iceberg or Delta Lake.
Currently working as a Machine Learning Engineer at Intella.
Interested in end-to-end Machine Learning pipelines, MLOps, Data Engineering and deploying production-ready scalable systems.
Previous experience in the field of Computer Vision where I’ve been able to publish a co-authored paper during graduation. During my academic experience at the University of Bari I’ve worked on the ISO 25010 certified project e-GLU BOX: a platform for usability tests made for PA. The application was developed both as a web application using Laravel and as a mobile app using Flutter.
Technical skills:
- Python, Pandas, Polars, Numpy, Pyspark, Dash, Prefect
- Delta Lake, Iceberg, DuckDB, Databricks, Azure
- Pytorch, Keras, Tensorflow, Sklearn
- MLFlow, SHAP, Docker
- Flutter, Laravel
Domains of interest:
- Artificial Intelligence, Machine Learning, Data Science, Time Series Forecasting, Computer Vision, Data Engineering