PackagingCon

Kevin Mittman

Kevin Mittman is a GNU/Linux enthusiast with a passion for automation. He is a system software engineer at NVIDIA, with a focus on the installer packaging and release process for CUDA, the NVIDIA driver, and other CUDA-X products. Before joining NVIDIA, Kevin began his career in the open source community, maintaining Debian packages for Maemo and later an ArchLinux-based kiosk Linux LiveUSB distro.


Session

11-10
17:50
20min
Triforce: repository management
Kevin Mittman

As repositories grow in size with packages, the time complexity starts to become O(n*log(n)) to keep the metadata up-to-date, because retaining the history requires re-parsing published packages and those must be available locally.

At NVIDIA, the Triforce repository management system handles the release process in O(n). To re-generate the metadata, one or more product release candidates are merged together using OverlayFS, on top of the public repository; this avoids the need for copying hundreds of gigabytes of existing packages, significantly reducing the I/O and storage usage.

Another consideration is how long it takes to build the metadata, by default generated from scratch each time. For RPM repositories, createrepo_c has the flag --update which skips over existing packages that have not changed. However for Debian repositories, existing tools such as apt-ftparchive lack such functionality. Comparing the filenames and file sizes is a good enough indicator if the package can be skipped. Parsing dpkg --info, it is possible to form the fields in a deterministic order for each block. From there it is as simple as appending the new metadata to the existing Packages.gz and regenerating the Release file.

Metadata & Building things
Room 4