Andre Jasiskis is a founding member of Nubank's data platform, currently working on Nubank's streaming platform. Has been thinking a lot about data ingestion, fast batch, and streaming processing, specifically on how to handle the exponential growth in consumers and data volume.
It is clearly beneficial for an organization to make data-driven decisions,
decentralize access to data processing and empower every team to generate valuable information.
There are many ways to achieve these goals, but in an environment of rapid growth, building an accessible Data Platform is just the first step. What happens next determines its long-term success or its dramatic demise.
In this presentation, we discuss the main perils of building a platform that
processes over 80000 unique datasets built by 1000 people across different
teams, how to avoid them, and where to go from there.