2019-09-16 –, Ferrier Hall
Software development is entering an era where the behavior of programs critically depends on the data they were trained on. In this setting, data is the new source code, and this opens the door to challenges like versioning and collaboration on numerical data. Hangar, git for your data
In this AI first world, Data is the new source code, and this opens the door to challenges like versioning and collaboration on numerical data. Enter Hangar, an open-source tool by [tensor]werk that brings Git-style version control to n-dimensional arrays. It supports versioning, branching, merging, time-travel, diffing, remote repositories and partial fetching, with data loaders for the major deep learning frameworks. At its core hangar is designed to solve many of the same problems faced by traditional code version control system (ie. Git), just adapted for numerical data:
- Time travel through the historical evolution of a dataset
- Zero-cost Branching to enable exploratory analysis and collaboration
- Cheap Merging to build datasets over time (with multiple collaborators)
- Completely abstracted organization and management of data files on disk
- Ability to only retrieve a small portion of the data (as needed) while still maintaining a complete historical record
- Ability to push and pull changes directly to collaborators or a central server (ie a truly distributed version control system)
The ability of version control systems to perform these tasks for codebases is largely taken for granted by almost every developer today; However, we are in fact standing on the shoulders of giants, with decades of engineering which has resulted in these phenomenally useful tools. Now that a new era of “Data-Defined software” is taking hold, we find there is a strong need for analogous version control systems which are designed to handle numerical data at large scale... Welcome to Hangar!
