Niels Neerhoff
Niels is a software engineer at Palm, the AI treasury startup. Simon is a data scientist and engineer, currently working as tech lead at ING Bank. Both have experience at the intersection of software engineering and data science within the fintech domain.
He/him
ING Bank
Data engineer
Session
Scaling data science pipelines in research and industry poses well-known maintainability challenges (big ball of mud). Research codebases must support rapid iteration as insights evolve, while industry systems must scale amid changing business needs and organizational complexity. Effective projects should remain maintainable without overhauling the entire code base for each change. Ideally, evolving from a notebook experiment to a production-grade application should feel natural, with minimal overhead.
In this talk, we show how data science projects benefit from established software engineering principles, particularly those inspired by functional programming, in Python. The first part of the talk outlines the design principles. The second part, will go into our (brutally honest) insights from applying these in various research projects, spanning from master student experiments to the applications in our R&D teams.