2021-11-10 –, Room 4
As repositories grow in size with packages, the time complexity starts to become O(n*log(n)) to keep the metadata up-to-date, because retaining the history requires re-parsing published packages and those must be available locally.
At NVIDIA, the Triforce repository management system handles the release process in O(n). To re-generate the metadata, one or more product release candidates are merged together using OverlayFS, on top of the public repository; this avoids the need for copying hundreds of gigabytes of existing packages, significantly reducing the I/O and storage usage.
Another consideration is how long it takes to build the metadata, by default generated from scratch each time. For RPM repositories, createrepo_c has the flag --update which skips over existing packages that have not changed. However for Debian repositories, existing tools such as apt-ftparchive lack such functionality. Comparing the filenames and file sizes is a good enough indicator if the package can be skipped. Parsing dpkg --info, it is possible to form the fields in a deterministic order for each block. From there it is as simple as appending the new metadata to the existing Packages.gz and regenerating the Release file.
Package repositories are a core component of most Linux distributions, providing a mechanism for installing and updating various software components. Package managers require an entry point such as repomd.xml (RPM) or Release (Debian) to discover software available in a repository and using properties stored in a manifest file, like *-primary.xml.gz (RPM) or Packages.gz (Debian) such as version, dependencies, and conflicts to build a transaction to execute.
For the CUDA repository, on a per distro / arch basis, everything is stored into one package repository. This is compounded as release candidates for independent products are validated in parallel, where product A may ship before product B, vice versa, or simultaneously. To handle these different scenarios, a train model is implemented in which one or more “passenger” RCs may be released during a given slot.
This talk will describe some of the challenges of releasing deliverables on a fast cadence, lessons learned, and finally provide examples of code used in production.
Kevin Mittman is a GNU/Linux enthusiast with a passion for automation. He is a system software engineer at NVIDIA, with a focus on the installer packaging and release process for CUDA, the NVIDIA driver, and other CUDA-X products. Before joining NVIDIA, Kevin began his career in the open source community, maintaining Debian packages for Maemo and later an ArchLinux-based kiosk Linux LiveUSB distro.