Quentin Lhoest
Open source developer in AI and head of datasets at Hugging Face. Quentin has been working on open source projects to democratize AI with a focus on data loading and sharing since 2020. His work includes helping research communities and companies in AI for text, audio, image, video and robotics, and also working on compute and storage solutions for AI datasets at Hugging Face.
Session
Deduplicating data means keeping only a single occurrence of every piece of data. Applied to cloud storage, it can significantly reduce physical file sizes, reduce storage costs and accelerate I/O. In this talk, I will cover how Data Engineers can use open-source tools to deduplicate Parquet data with Parquet Content Defined Chunking (CDC) and how it works under the hood (minutes 0-10). I'll also present simple examples as well as real-world examples from the 100PB+ of datasets and buckets on Hugging Face (minutes 10-20).