1.11 packagingcon-2021 2021-11-09 2021-11-10 2 00:05 https://pretalx.com https://pretalx.com/media/packagingcon-2021/img/pkgcon_osNlIPy.png UTC Plenaries Lightning Talk 2021-11-09T21:15:00+00:00 21:15 00:02 Python packages are the fundamental units of shareable code in Python. Packages make it easy to organize, reuse, and maintain your code, as well as share it between projects, with your colleagues, and with the wider Python community. Despite their importance, Python packages can be difficult to understand and cumbersome to create for beginners and seasoned developers alike. Fortunately, packaging tools exists to streamline the packaging process. This lightening talk discusses an accessible and practical approach to creating packages using modern and mature tools such as poetry, cookiecutter, pytest, sphinx, GitHub, and GitHub Actions! packagingcon-2021-13642-a-practical-and-modern-approach-to-python-packaging /media/packagingcon-2021/submissions/HPHEMU/hex_NfKAeIe.jpeg Tomas Beuzen en Disclaimer: this talk is a 2-minute abridged version of our open-source book _Python Packages_ (https://py-pkgs.org). false https://pretalx.com/packagingcon-2021/talk/HPHEMU/ https://pretalx.com/packagingcon-2021/talk/HPHEMU/feedback/ Plenaries Lightning Talk 2021-11-09T21:20:00+00:00 21:20 00:02 Ada is a venerable language with a long and proven trajectory mainly in embedded and critical systems. With a small but close-knit Open Source community, Ada has lacked a package manager until recently. Alire (Ada Library Repository, https://alire.ada.dev/) is a package manager for the language that supports the GNAT Ada compiler, available through the FSF as a GCC frontend. This lighting talk aims to introduce Alire to the family of package managers and give a few highlights of its characteristics. packagingcon-2021-12140-alire-the-ada-package-manager Alejandro R. Mosteo en false https://pretalx.com/packagingcon-2021/talk/S78FQA/ https://pretalx.com/packagingcon-2021/talk/S78FQA/feedback/ Plenaries Lightning Talk 2021-11-09T21:25:00+00:00 21:25 00:02 Bash is known for being a quirky language, mainly used to glue different programs together in small scripts. As a result of this perception (and partly due to a lack of language features), Bash has a weak library ecosystem. All things considered, this makes it difficult to find and integrate Bash code that is both robust and devoid of platform-specific hacks. I wish to solve this predicament by proposing a Bash package manager called [Basalt](https://github.com/hyperupcall/basalt). It standardizes and substantially simplifies the problem of code reuse across Bash projects. Basalt is defining what it means to create a “Bash library” and a “Bash application”; it is also enabling the emergence of cutting-edge Bash libraries, such as complete TOML parsers. packagingcon-2021-12362-bash-comprehensive-dependency-management Edwin Kofler en Basalt is a Bash package manager that makes it dead simple to download and reuse shell scripts and Bash libraries. It can resolve dependencies for a particular Bash project, or globally for a particular user. Its CLI and configuration interface is inspired by both Yarn and Cargo. Basalt is breaking the mold of what it means to create a Bash application. Not only is it enabling the emergence of TOML parsers written in pure Bash, but it is also making it straightforward to improve performance by making dynamically loadable custom builtins a first-class feature. In this lightning talk, I quickly explain the problems of code reuse in Bash projects, and explain how Basalt solves various use cases. Then, I provide a birds-eye view on the internals of Basalt and wrap up the talk by mentioning what’s next on the roadmap. false https://pretalx.com/packagingcon-2021/talk/J9SYFM/ https://pretalx.com/packagingcon-2021/talk/J9SYFM/feedback/ Plenaries Lightning Talk 2021-11-09T21:30:00+00:00 21:30 00:02 An brief intro to the data behind github's dependabot tool and how it may be useful to package maintainers. packagingcon-2021-12122-github-advisory-database-the-database-backing-dependabot Jon en false https://pretalx.com/packagingcon-2021/talk/MTQARP/ https://pretalx.com/packagingcon-2021/talk/MTQARP/feedback/ Plenaries Lightning Talk 2021-11-09T21:35:00+00:00 21:35 00:02 To a newbie in the packaging world, writing recipes could seem quite intimidating. Even people who are not so new would agree that writing package recipes is tiresome, not to say highly errorprone. Example recipes and templates help, but one would rather their package recipe was generated automatically and was perfectlyconcise. Of course, Anaconda provides Conda Skeleton. Although Conda Skeleton is a helpful tool, it falls short of being the perfect recipe generator for several reasons: it's slow in generating recipes, cannot be deployed on systems without conda, andhas a huge number of dependencies. The recipes itgenerates are also not always concise. Grayskull solves all these problems. Grayskull is an automatic conda recipe generator. It generates concise conda recipes for Python packages available on PyPI specially customized for (but not limited to) the conda-forge ecosystem. Grayskull significantly improves upon existing recipe generators in terms of speed, conciseness of the recipes, packaging environment specificity, and memory usage. Grayskull has proved to be an extremely useful tool for the packaging ecosystem by generating accurate recipes quickly. Grayskull, by making it possible to generate conda recipes for PyPI packages, brings PyPI closer to the Conda and reduces fragmentation inthe packaging ecosystem. packagingcon-2021-13118-kickstart-your-journey-into-the-conda-packaging-world-with-grayskull Mahe Iram Khan en After introducing Grayskull, I will talk in detail about some of its inner workings that make it faster and more efficient than existing recipe generators: - Grayskull generates recipes taking in consideration the platform, Python version available, selectors, compilers (Fortran, C and C++), package constraints, license type etc. - It uses metadata available from multiple sources to create the best recipe possible. - In the case of noarch: python, Grayskull is smart enough to detect when the recipe supports it, which is not done by Skeleton. It is important to highlight that Skeleton does not detect compilers either. Whereas Grayskull always tries to detect them. - The dependencies of Grayskull are quite reduced when compared to Conda Skeleton. Conda Skeleton relies on conda which is a huge project, therefore Conda Skeleton has a lot of dependencies. Grayskull is a standalone application which does not rely on conda. It can be easily deployed on systems without conda. It is pip installable. - Conda Skeleton creates a separate conda environment when it tries to generate the recipe and it takes a lot of time because it also runs the solver. Grayskull, on the other hand, creates a small and temporary virtual environment to simulate the installation of the package using the source tarball for Python projects. - Conda Skeleton sometimes mixes some dependencies and generates a quite "fat" recipe. Grayskull does not. I will demonstrate live how easy it is to generate recipes with Grayskull (really, it’s just a single command: `grayskull pypi <package-name>` ). I will also demonstrate how to use the online version of Grayskull. After the live demonstration, I will talk about the features that Grayskull is presently missing; the addition of which will make Grayskull an even more versatile tool for recipe generation. I will also discuss the work I have been doing on Grayskull via my internship at Quansight Labs under the mentorship of Jaime Rodríguez-Guerra and Vinicius D. Cerutti: - Presently Grayskull generates recipes for Python packages available on PyPI. I am working on adding more package origins to Grayskull; ability to generate recipes for packages available only as Github, Gitlab repositories, ability to generate recipes for PyProject packages. - The next version of Grayskull could have the ability to generate recipes for R packages available on CRAN. - And the next could have the ability to generate recipes for C++ packages. It also has an online version: https://www.marcelotrevisani.com/grayskull false https://pretalx.com/packagingcon-2021/talk/FRJACA/ https://pretalx.com/packagingcon-2021/talk/FRJACA/feedback/ Plenaries Lightning Talk 2021-11-09T21:40:00+00:00 21:40 00:02 Packaging and publishing software remains a challenge for many researchers. Here, we present the "[Packaging and Publishing with Python](https://carpentries-incubator.github.io/python-packaging-publishing/)" lesson from the [Carpentries Incubator](https://carpentries-incubator.org/). The Carpentries Incubator is a [The Carpentries](https://carpentries.org/) initiative for community-developed lessons. Lessons can be taught in workshops in both online and in-person formats, and can also be used for self-guided study. In this lightning talk, we are going to go over what the lesson covers, how you can teach it and how to contribute to it. Finally, we are going to demonstrate how learning to package software is a useful skill for researchers, and how this lesson supports that. packagingcon-2021-12102-python-packaging-and-publishing-carpentries-incubator-lesson Vinícius Salazar en false https://pretalx.com/packagingcon-2021/talk/AWN3MG/ https://pretalx.com/packagingcon-2021/talk/AWN3MG/feedback/ Plenaries Lightning Talk 2021-11-09T21:45:00+00:00 21:45 00:02 An introduction to the current state of software bootstrapping and defenses against the trusting trust attack. packagingcon-2021-12671-stage0-bootstrapping-trust Jeremiah Orians en false https://pretalx.com/packagingcon-2021/talk/E3M8HG/ https://pretalx.com/packagingcon-2021/talk/E3M8HG/feedback/ Plenaries Lightning Talk 2021-11-09T21:50:00+00:00 21:50 00:02 Every project has installation instructions describing system requirements as a list of system and other packages to install. It is time to get rid of this README section! Let's look at the rise of universal package management where one tool and one unified spec can rule them all. packagingcon-2021-13293-the-rise-of-mostly-universal-package-managers /media/packagingcon-2021/submissions/ZFJUEJ/kitten_JeoW98j.jpg Philippe Ombredanne en Because **no tech stack is an island** with a single programming language and packaging ecosystem, universal package management tools that abstract the kinks and subtle differences between package ecosystems are emerging to solve a practical problem all package authors and users are facing. Let's look at what they are, and what they could become, and why you need one. false https://pretalx.com/packagingcon-2021/talk/ZFJUEJ/ https://pretalx.com/packagingcon-2021/talk/ZFJUEJ/feedback/ Plenaries Lightning Talk 2021-11-09T21:55:00+00:00 21:55 00:02 [Turing.jl](https://github.com/TuringLang/Turing.jl) is a Julia library focusing on Bayesian inference with probabilistic programming. It has a special focus on modularity, and it decouples the modelling language and inference methods. This talk highlights the features of Turing.jl. Furthermore, references are provided to tutorials for working with Turing. Slides of this talk are available at [bit.ly/turing-an-overview](https://docs.google.com/presentation/d/e/2PACX-1vQ5WvGcgLnkiNh004BWGtbUeNnV54vQ4ypFpsD7DJfm9eX6jrFyf5EIPAPB-0ZdorSt9-I6OdNx5GKF/pub?start=false&loop=false&delayms=3000&slide=id.p) and also available on the [GitHub repository](https://github.com/SaranjeetKaur/PackagingCon2021_Slides_Lightning_Talk/). packagingcon-2021-13634-turing-jl-an-overview Saranjeet Kaur Bhogal en false Slides https://pretalx.com/packagingcon-2021/talk/QL7K9K/ https://pretalx.com/packagingcon-2021/talk/QL7K9K/feedback/ Plenaries Lightning Talk 2021-11-09T22:00:00+00:00 22:00 00:02 There are two predominant models for software updates: the package management approach, which resolves new sets of compatible software to install together and respects dependency declarations, and the "update channel" approach, where an installed software component subscribes itself to updates via a stream of external metadata (i.e. Google Chrome's update model). The Operator Lifecycle Manager for Kubernetes combines both approaches: software packagers can provide valid update graphs for their components in addition to dependency information, and the on-line solver considers both when selecting and installing packages. packagingcon-2021-12412-unifying-update-channels-and-dependency-resolution Evan Cordell en false https://pretalx.com/packagingcon-2021/talk/JLXJBV/ https://pretalx.com/packagingcon-2021/talk/JLXJBV/feedback/ Room I Talk 2021-11-09T16:00:00+00:00 16:00 00:50 Wolf will welcome everyone and say a couple of words about PackagingCon and how we are going and how the virutal conference is going to work ## Keynote Todd Gamblin, Steven! Ragnarök and Matthias Meschede are going to talk about "The Taxonomy of Package Managers" – expect a fun talk about the history of package management and an overview of the different species of package managers out there packagingcon-2021-13657-welcome-keynote Wolf VollprechtTodd GamblinSteven! RagnarökMatthias Meschede en false https://pretalx.com/packagingcon-2021/talk/KUFQGD/ https://pretalx.com/packagingcon-2021/talk/KUFQGD/feedback/ Room I Talk 2021-11-09T17:00:00+00:00 17:00 00:20 We have developed a system which will automatically generate packages for deb based packaging systems such as Debian and Ubuntu, RPM based packaging systems such as Fedora and RHEL, as well as source based packaging/distribution systems such as Gentoo or OpenEmbedded. This talk will delve into how and why we’ve done it. We will cover lessons learned over the course of more than ten years of experience and then discuss where we’re going next and what tools and approaches we’ve developed that others may find useful. packagingcon-2021-12131-automated-packaging-for-multiple-platforms-successes-and-lessons-learned-while-packaging-ros Metadata & Building things Steven! RagnarökTully Foote en The ROS project has been generating packages for several platforms for over 10 years. The toolchain continues to grow and evolve. We currently have support for Debian, Ubuntu, Fedora, RHEL, OpenEmbedded, Gentoo, as well as are working on Conda, there are other community based efforts ongoing. We have targeted amd64, i386, armhf, arm64 architectures. We’ve operated 17 distributions with which target between 3 and 5 platforms officially. The 5 currently active distributions contain 6241 packages of which 3057 are unique by package name. 1132 people have contributed to our release repository including dependency data and there are 687 unique maintainer emails in our publicly listed source packages. And all these packages are downloaded many millions of times per month. There’s many moving parts and many different communities to bring together to make all this happen regularly. Our needs and goals are sometimes at odds with upstream policies but we still have found ways to work together. This talk will be an overview of where we’re coming from with our needs and requirements, followed by how we’re achieving that. We’ll talk about existing tools and processes that we’re following from existing projects, and then also about how we’ve chosen to diverge and why we chose to do so as well. And provide a quick overview of the various tools that we’ve developed, focusing on those that are potentially reusable outside the ROS context. false https://pretalx.com/packagingcon-2021/talk/K7LDFB/ https://pretalx.com/packagingcon-2021/talk/K7LDFB/feedback/ Room I Talk 2021-11-09T17:25:00+00:00 17:25 00:20 We are offering Packit, a free GitHub app and GitLab integration which enables you to build and test your upstream project on an RPM-based Linux distribution like Fedora Linux, CentOS Stream, Mageia or openSUSE. Once you get RPM builds of your project, you can be pretty sure that your project will work once released and delivered via the downstream distribution. The core functionality of Packit is built around pull requests (as a standard CI system) and releases (bring the release to Fedora rawhide). You can read more about Packit at https://packit.dev/ In this session, Franta and Tomas will describe the Packit project, Fedora’s packaging workflow, showcase some of the well-known projects which use Packit and offer a brief perspective on what it’s like to develop and maintain the integration service. packagingcon-2021-12191-integrating-upstream-projects-to-fedora-linux Metadata & Building things /media/packagingcon-2021/submissions/GY99JV/logo-square-small-borders_E0qgtqQ.jpg František LachmanTomas Tomecek en false Slides https://pretalx.com/packagingcon-2021/talk/GY99JV/ https://pretalx.com/packagingcon-2021/talk/GY99JV/feedback/ Room I Talk 2021-11-09T17:50:00+00:00 17:50 00:20 An overview of the policies, design choices, and tooling that allow a team to maintain the Homebrew ecosystem, enabling timely delivery of updates while minimizing regressions in packages and dependency trees. packagingcon-2021-12158-force-multipliers-in-package-management-how-homebrew-maintainers-keep-up-with-10-000-packages Metadata & Building things Caleb Xu en In this talk, we'll explore some of the things that allow the Homebrew maintainer team to tackle day-to-day maintenance tasks, leaving time and energy to address bigger things. We'll visit some of the technical aspects of the Homebrew ecosystem that make this possible as well as some of the policies and culture that help prevent maintainer burnout. false https://pretalx.com/packagingcon-2021/talk/NRGCVG/ https://pretalx.com/packagingcon-2021/talk/NRGCVG/feedback/ Room I Talk 2021-11-09T18:15:00+00:00 18:15 00:20 Bitnami is an application packaging and publishing startup that was acquired by VMware in 2019. It is a leading provider of prepackaged open source software that runs natively in environments where a large portion of developers and other users want to build or deploy applications in the major public clouds, on laptops, and on Kubernetes. Over the last few years with the increased popularity of containers and platforms like Kubernetes, Bitnami's growth has raised exponentially and several of its containerised applications are now well over +1B downloads each. The secret sauce for Bitnami success has always been trying to make Open Source safe and easy to use. Sounds simple, but it is actually very challenging. A robust pipeline must be able to build many different flavours of open source software targeting many different operating systems and clouds, and it has to be simple. Abstracting users from complexity. Additionally, Bitnami focuses on making Open Source safer by having those application packages running within a continuous update loop taking care of releasing updates when new vulnerabilities or attacks are found. In this talk we would like to go over how we have made this possible over the last 15 years. packagingcon-2021-12114-bitnami-15-years-bringing-open-source-to-the-masses Metadata & Building things Martin Perez, Beltrán RuedaBeltrán Rueda en false https://pretalx.com/packagingcon-2021/talk/7MQPPU/ https://pretalx.com/packagingcon-2021/talk/7MQPPU/feedback/ Room I Talk 2021-11-09T19:00:00+00:00 19:00 00:20 The Julia programming language features a built-in package manager commonly referred to as "Pkg". It's actually the third iteration of package manager for the language, code-named Pkg3 while in development. The previous iterations were quite traditional, inspired by Perl's CPAN and RubyGems. Pkg3 is *different*. This talk explores how it differs from its predecessors and other package managers and what lessons we've learned while developing it and scaling up its usage. packagingcon-2021-12018-julia-s-pkg-design-rationale Deep Dives /media/packagingcon-2021/submissions/K8GPRA/19329590_sAUg9JO.jpg Stefan Karpinski en Some salient features of Julia's Pkg that will be covered in this talk: - Packages are identified by globally unique UUID, not just name. This allows different packages with the same name to co-exist in the dependency graph of a project. Names used in source code are mapped to UUIDs in a project-local `Project.toml` file which also contains other project metadata. - Code loading works by looking up the cryptographic hash of the source tree of a specific version of a package in a project-local `Manifest.toml` file. This source hash is used to look up the path where the code should be loaded from. Since each package version is identified and found by tree hash, its content can always be checked for correctness and caches never need to be invalidated. Manifest files can be tracked in version control providing perfect reproducibility by default. - It's completely normal for mulitple versions of the same package to be installed at the same time, used by different projects. This is kind of like Python virtual environments but built into the language, with common versions shared, and without requiring any environment variable tricks. Pkg has a `gc` command that searches through known manifest files and garbage collects (i.e. deletes) any package versions that are no longer in use anywhere. - It's not just Julia source packages that are immutable and content-addressed: Pkg also installs libraries and other binary dependencies as immutable, content-addressed tarballs of pre-compiled, system-specific file trees. The right variant for a given operating system / libc version / libc++ version (etc.) is chosen and installed, but that combination is pre-built and simply needs to be downloaded and put in the right place. This makes installing binary dependencies incredibly fast and reliable. It also provides tremendous benefits for reproducibility since all of this is cryptographically hashed, content-addressed, immutable, tracked in project-local version control, and persisted forever by the global network of package servers. - Pkg has a federated package registry system. There is a general public registry that Julia clients get their packages from by default, but other registries can be added and used alongside it. It is common for companies and research labs to have their own private and/or public registries of packages. The use of UUIDs to identify packages even makes transitioning a package from private to public extremely smooth. It's even possible for some versions of a package to be public while others—older or newer—remain private. - UUIDs provide some protection from [dependency confusion attacks](https://medium.com/@alex.birsan/dependency-confusion-4a5d60fec610), but this depends on UUIDs remaining secret which they are not designed to be. The same features that facilitate migrating a package from private to public inadvertently allow dependency confusion attacks. The General registry allows submission of lists of private UUIDs to block from registration, but this is a stopgap measure at best. Better solutions to this common packaging ecosystems problem are sought. false https://pretalx.com/packagingcon-2021/talk/K8GPRA/ https://pretalx.com/packagingcon-2021/talk/K8GPRA/feedback/ Room I Talk 2021-11-09T19:25:00+00:00 19:25 00:20 Homebrew is a free and open-source package manager, initially written for macOS. Linuxbrew, a fork of Homebrew for Linux, was created in 2012. In 2019, we announced the official support for Linux and Windows 10 (with Windows Subsystem for Linux). The Linux-specific code of the package manager was back-ported from Linuxbrew to the main Brew repository in 2018/2019. But the story did not end there. The Linux packages were still living in a separate repository: linuxbrew-core. We had to migrate all the changes from the Linux repository to the main repository (homebrew-core). There were more than 5000 lines of code to be back-ported. We also started building Linux packages in homebrew-core, so we had to set up Linux CI along the existing macOS one. As this task is now almost completed and we will soon decommission linuxbrew-core, I would like to come back on the details of this epic migration. This talk will make a small retrospective on why it took us almost 2 years to finish the migration. I will also take the opportunity to discuss the setup of our Linux CI, and the issues we faced while doing so. packagingcon-2021-12121-homebrew-improved-linux-support-and-a-historical-review-of-our-linux-ci Metadata & Building things Michka Popoff en The talk will focus on a few topics related to the migration from linuxbrew-core to homebrew-core. I will go through the way the linuxbrew-core repository co-existed with homebrew-core repository over the years, and why we needed to decommission linuxbrew-core. Keeping linuxbrew-core in sync with homebrew-core has drained a lot of energy out of multiple maintainers, and had become too complex. The current workflow also often broke packages for our Linux users, which was not acceptable. Ending the migration will give us time to finally focus on more interesting tasks and new features, help triage more issues and help our users more effectively. I will also discuss different CI solutions we have used over the years to build binary packages: Docker hub, Travis, Circle.ci, Azure pipelines and finally GitHub Actions. As we initially did not have any funding for CI, we had to rely on free tiers, which caused a higher workload for maintainers, as a lot of manual intervention was needed. We even built some packages with our personal hardware when necessary. One last topic I want to discuss is open source maintainer bandwidth, and why it took us two years to finalise the linuxbrew-core to homebrew-core migration. The number of packages to maintain, the number of maintainers, and more importantly the number of maintainers willing to do ops and fix things in the package manager itself will be discussed. Also, I will have a quick look at the financial aspect when it comes to setup CI for a project as big as Homebrew. false https://pretalx.com/packagingcon-2021/talk/KTNQRB/ https://pretalx.com/packagingcon-2021/talk/KTNQRB/feedback/ Room I Talk 2021-11-09T19:50:00+00:00 19:50 00:20 In the past 30 years or so of widespread code reuse, programming language communities have come up with various approaches to solving problems of code reuse. These efforts are often developed in isolation, leading to a divergence in concepts and terminology. What can we learn from one another? And how can we use this understanding to make better tools for managing software dependencies? packagingcon-2021-12060-putting-concepts-into-boxes-a-survey-of-packaging-systems-and-patterns-of-code-reuse Metadata & Building things Mattt en A native speaker of a language can communicate automatically without giving a second thought about how they do it. Likewise, a software developer who works exclusively in an ecosystem can be quite productive without really understanding how their code gets turned into programs. When you learn another language, you gain completely different kind of understanding. You start to reason about concepts formally rather than intuitively. You discover that ideas that you took for granted in one may be radically different (or missing entirely) in another. You come away with knowledge of the new, a new perspective of the familiar, and an appreciation for the unknown. In this talk, we'll take a look at the landscape of packaging systems. We'll attempt to identify and formalize concepts they share in common, and distinguish the incidental and inherent differences among them. By doing so, we hope to provide useful models for building tools within and across software ecosystems. true https://pretalx.com/packagingcon-2021/talk/YQSMZQ/ https://pretalx.com/packagingcon-2021/talk/YQSMZQ/feedback/ Room I Talk 2021-11-09T20:15:00+00:00 20:15 00:20 If you're managing cloud native applications, you already have a reliable, secured, performant container registry across your development to production environments. Where will you store your Helm charts, OPA Bundles, WASM, SBOMs, Scan Results, GitOps/RegOps and deployment artifacts? Do you really want to stand up and manage Yet Another Storage Solution (YASS)? Should you pull your developer focused Git infra into production? OCI Artifacts expands container registries to store any artifact. Artifacts are now adding Reference Types to store a graph of objects, including SBOMs, Signatures, Security Scan Results. We'll review the journey for OCI Artifacts and how you can build a new cloud native thing, without having to build and maintain YASS. packagingcon-2021-11953-oci-artifacts-using-container-registries-for-any-cloud-native-artifact Registries Steve Lasker en false https://pretalx.com/packagingcon-2021/talk/ZCRDEQ/ https://pretalx.com/packagingcon-2021/talk/ZCRDEQ/feedback/ Room I Talk 2021-11-09T20:40:00+00:00 20:40 00:20 This talk discusses the current implementation of package registries for the Julia package manager and some of the lessons learned along the way. packagingcon-2021-12111-package-registries-for-the-julia-package-manager Registries Kristoffer Carlsson en A package registry is a collection of metadata for a set of packages. Among other things, this includes: - what versions are available for each package - what dependencies each package version - what package versions are compatible with other package versions This information is used by the Julia Package manager's "resolver" when a package operation is performed (e.g. adding/updating packages). The task of the resolver is to return a set of package versions with "as high version as possible" under the constraints that all those versions are compatible with each other. This talk discusses some of the lessons we have learned about package registries from our experiences in developing the Julia package manager over many years. This includes considerations like: - what file format to use - support for multiple registries - how to download/update the registry - performance considerations w.r.t the size of the registry/number of files false https://pretalx.com/packagingcon-2021/talk/QNJEMW/ https://pretalx.com/packagingcon-2021/talk/QNJEMW/feedback/ Room 2 Talk 2021-11-09T17:00:00+00:00 17:00 00:20 A deep-dive on the interesting (both good and bad) aspects of the Homebrew package manager that will be interesting to other package manager maintainers or enthusiasts. packagingcon-2021-12009-homebrew-a-packagers-deep-dive Deep Dives Mike McQuaid en Based on my experience as a user (and very sporadic packager) of other OS system and language package managers I'll detail the things I feel that Homebrew does well, badly and what we plan on changing and what we cannot. false https://pretalx.com/packagingcon-2021/talk/JPXYSD/ https://pretalx.com/packagingcon-2021/talk/JPXYSD/feedback/ Room 2 Talk 2021-11-09T17:25:00+00:00 17:25 00:20 One of the challenges in HPC is to deliver a consistent software stack that balances the needs of the system administrators with the needs of the users. This means running recent software on enterprise Linux distributions that ship older software. Traditionally this is accomplished using environment modules, that change environment variables such as $PATH to point to the software that is needed. At Compute Canada we have taken this further by distributing a complete user-level software stack, including all needed libraries including the GNU C library (Glibc), but excluding any privileged components. Our setup combined Nix, and now combines Gentoo Prefix for the bottom layer of base components, EasyBuild for the top layer of more scientifically inclined components, Lmod to implement environment modules, and the CernVM File System (CVMFS) to distribute it to Canadian supercomputers and anyone else who is interested. This approach has gained interest in other places, most notably with the EESSI project that originated in Europe. I will describe our setup and discuss the pros and cons of Nix versus Gentoo Prefix, and the challenges that come with using glibc in a non-standard location. packagingcon-2021-12120-combining-cvmfs-nix-or-gentoo-prefix-lmod-and-easybuild-at-compute-canada Deep Dives Bart Oldeman en false https://pretalx.com/packagingcon-2021/talk/YWRVCT/ https://pretalx.com/packagingcon-2021/talk/YWRVCT/feedback/ Room 2 Talk 2021-11-09T17:50:00+00:00 17:50 00:20 Nix has awesome packing tools. Many of them. Windows was always the landscape of Next -> Next -> Next. Ever wished you could take all of those Windows applications you run, install them, and not have to click anything? Easily keep them up to date and not click anything? And in WINDOWS? Come with me on this journey, and you’ll see a world of Windows Automation, Package Management and a thriving Community. packagingcon-2021-12194-what-is-chocolatey-the-package-manager-for-windows Deep Dives Paul Broadwith en When we think about package management we always think about nix - apt, yum, dnf etc. Widnows doesn't really come to mind. I want to introduce the audience to the package manager for Windows; Chocolatey. Chocolatey has been around since 2011 and has a Community Package repository of over 800 unique packages, 100K+ package versions and in excess of 1B package downloads. false https://pretalx.com/packagingcon-2021/talk/LMNF3H/ https://pretalx.com/packagingcon-2021/talk/LMNF3H/feedback/ Room 2 Talk 2021-11-09T18:15:00+00:00 18:15 00:20 Package management is the vital tool enabling reuse of other's code from around the world. However, this dream quickly collides with business fundamentals such as security, reliability, and authenticity. In this talk, we'll discuss vcpkg's new asset caching capabilities and how they enable enterprises to participate in the open source community without compromising essential objectives -- especially for secured networks without internet access. packagingcon-2021-12265-vcpkg-asset-caching-solving-the-air-gap Registries Robert Schumacher en This presentation will cover the asset caching functionality in vcpkg which enables uniform mirroring of external assets including sources and prebuilt tools. We'll cover how this system works in conjunction with vcpkg's registry, port, and binary caching systems to enable offline and disconnected systems to still take full advantage of open source projects and recipes without modification. false https://pretalx.com/packagingcon-2021/talk/QQEVAK/ https://pretalx.com/packagingcon-2021/talk/QQEVAK/feedback/ Room 2 Talk 2021-11-09T19:00:00+00:00 19:00 00:20 When performing dependency resolution, a package manager makes choices about which versions of packages to install. These choices impact the final bundled application in a variety of ways, such as: correctness, code size, performance and security vulnerabilities. Different production package managers (such NPM, Pip and Cargo) can produce very different results when resolving identical lists of dependencies, which can lead to users being confounded and having little choice over dependency resolution behavior. We address this by developing a unifying formal model of the semantics of dependency resolution, and show that this model can encompass and highlight the key differences between NPM, Pip and Cargo. Further, our formal model delineates a design space of hypothetical package managers, which popular package managers only inhabit a part of. We enable empirical exploration of this design space by implementing MinNPM, a drop-in replacement for NPM which allows for user-specified customization of the dependency resolution semantics. Using MinNPM we explore the empirical differences within the design space, both among existing package managers' semantics, and with novel semantics which allow us to directly minimize arbitrary optimization objectives. packagingcon-2021-12257-minnpm-customizable-optimizing-dependency-resolution Solvers Donald Pinckney en false https://pretalx.com/packagingcon-2021/talk/K8C9HR/ https://pretalx.com/packagingcon-2021/talk/K8C9HR/feedback/ Room 2 Talk 2021-11-09T19:25:00+00:00 19:25 00:20 Most package managers need a dependency solver, but dependency solving is an NP-hard problem, and writing a correct solver from scratch is difficult to do correctly, let alone a fast solver. Simply understanding the solution space is a challenge, from simple SAT solvers, to specialized solutions like PubGrub and libsolv, to Satisfiabilty Modulo Theories (SMT) and Answer Set Programming (ASP) solvers. Solvers may need to optimize for multiple objectives -- preferring the most recent versions of dependencies is common, but multi-valued build options, optional dependencies, virtual dependencies, and build options like compilers, architectures, and ABI compatibility can also factor into a solve. We have recently shipped a new solver in the Spack package manager that relies on the `clingo` Answer Set Programming (ASP) framework to accomplish many of these goals. We'll talk about how we handle complex features like optional dependencies, generalized conditions, virtual dependencies (interfaces), compiler selection, ABI options, and multiple optimization criteria in around 500 lines of declarative code. We'll talk about some of the semantics of ASP that lend themselves to very general package solving (vs other models like SMT). Finally, we'll show some performance numbers with large package repositories. packagingcon-2021-12184-beyond-version-solving-implementing-general-package-solvers-with-answer-set-programming Solvers Todd Gamblin en Spack recently gutted its package solver and replcaed it with a very general solver based on ASP. ASP is a logic programming framework that borrows Prolog's syntax for first-order logic, but boils it down to SAT underneath. Over the past 25 years or so, ASP has made great strides in solver performance by borrowing from industrial SAT solvers and optimization tools, and ASP frameworks are able to solve much larger and more complex problems than most in the packaging domain. Spack's dependnecy model is targeted *both* at configuring from-source builds and at reusing optimized binary packages, but doing both of these things requires a much more general solver framework that is offered by most systems. In particular, the type of decisions handled by systems like PubGrub typically extends only to version selection, but to enable solvers to deal with build-time parameters requires much deeper package parameterization. In particular, Spack packages can depend on particular build options, compilers, compiler flags, and ABI options. The solver will reuse a binary *if* it meets particular build criteria, but it will decide instead to build a new version from scratch if it can't find a suitable binary. Builds can thus be "configured" by the solver for compatibility with selected binaries. Most package ecosystems and distributions enforce consistent ABI choices (like compiler and package ABI versions) -- our goal is to avoid these restrictions and allow users to write very general packages that can more easily build new stacks in exotic environments (like HPC machines). The talk will give an overview of Spack's dependency model, and we'll show how ASP and multiple optimization criteria are *needed* to implement these more general semantics. We'll compare to other package solvers like PubGrub and show how features of ASP solvers (like unsatisfiable cores) can be used to construct meaningful error messages even with complex solves. We'll also show the maintenance benefits of relying on an established solver framework -- the core logic of the solve can be implemented in around 500 lines of declarative code, which makes it much more maintainable than a custom, domain-specific solution. false https://pretalx.com/packagingcon-2021/talk/LJ9HJK/ https://pretalx.com/packagingcon-2021/talk/LJ9HJK/feedback/ Room 2 Talk 2021-11-09T19:50:00+00:00 19:50 00:20 Dependency solving is a hard problem, especially when mixed with additional features such as optional dependencies, multiple versions or availability of pre-releases. We present a rewrite from scratch of a recent algorithm called PubGrub, as a Rust library aiming at great performance and flexibility for reuse. We will dive into its core mechanisms, its high-level usage, as well as our new ideas enabling behavioral extensions such as optional dependencies, entirely in user space without changing the library API. packagingcon-2021-11955-building-a-flexible-dependency-solver-in-rust Solvers Matthieu PizenbergJacob Finkelman en false slides of the talk https://pretalx.com/packagingcon-2021/talk/MXVTEA/ https://pretalx.com/packagingcon-2021/talk/MXVTEA/feedback/ Room 2 Talk 2021-11-09T20:15:00+00:00 20:15 00:20 This talk will introduce some elements of ongoing research in the mathematical structure of package dependencies. This work helps to explain how to think about dependencies, how to compare expressiveness of dependency systems (and strength of solvers), and also how to model an algebra of operations of package repositories. packagingcon-2021-11918-an-invitation-to-order-theoretic-models-of-package-dependencies Solvers Gershom Bazerman en This will be an accessible talk introducing and motivating some of the more basic mathematical constructions in "The Semantics of Package Management via Event Structures" (https://arxiv.org/abs/2107.01542) and "The Topological and Logical Structure of Concurrency and Dependency via Distributive Lattices" (https://arxiv.org/abs/2004.05688). Among other things it will explain how the underappreciated Bruns-Lakser completion gives insight into nix-style package management, and how models of package repositories can draw on tools developed for the semantics of concurrent programs. This work both draws on modern mathematical techniques and also hands-on experience as a contributor and advisor to the Cabal package system in Haskell, as well as a maintainer of the Hackage repository of Haskell packages. false https://pretalx.com/packagingcon-2021/talk/PCC9GD/ https://pretalx.com/packagingcon-2021/talk/PCC9GD/feedback/ Room 2 Talk 2021-11-09T20:40:00+00:00 20:40 00:20 As a C and C++ developer how do you choose the right package management system for your code? There are a ton of questions that you should be asking yourself: does it have integrations, do we need end-to-end binary management, can it work with different software systems, will it provide consistency to my CI/CD workflow? Fortunately we have an open source solution that solves the riddle of package managers… Conan! Conan the Barbarian is forced to solve “The riddle… of steel,” so that he can reach his end goal of resting in eternity in Valhalla. To a somewhat lesser degree we want to make our users happy and solve the riddle of package managers and for us that is Conan with Artifactory. In this session we will talk about how C and C++ developers that are having issues when trying to create a repository system for their packages can solve this complex problem with Conan. Conan abstracts away build systems, defines a “Project API” for C++ project, provides a repository system for multi-binary packages, and serves as a building block for Continuous Integration workflows. packagingcon-2021-11956-the-riddle-of-package-managers-solved-by-conan Deep Dives /media/packagingcon-2021/submissions/YX8QCA/conan_OnWMD1x.jpeg Batel Zohar en In this session we will talk about how C and C++ developers that having some issues when trying to create a repository system for their packages while focusing on the open source project called Conan, Conan abstracts away build systems, defines a “Project API” for C++ project, provides a repository system for multi-binary packages, and serves as a building block for Continuous Integration workflows. false https://pretalx.com/packagingcon-2021/talk/YX8QCA/ https://pretalx.com/packagingcon-2021/talk/YX8QCA/feedback/ Room 3 Talk 2021-11-09T17:00:00+00:00 17:00 00:20 In this talk, Joshua Lock and Marina Moore will discuss common attacks on package managers, and the kinds of threats that package managers face as part of the software supply chain. They will then present The Update Framework (TUF), a mechanism for securing package managers against these threats in a simple, resilient way that will protect users against even nation state attacks. Package managers can adopt all features of TUF wholesale, or start with the subset that will be most helpful for their users. This talk will conclude with a demonstration of TUF’s versatility; explaining how TUF has been adopted by the Python Packaging Index (PyPI) to provide end-to-end protection of packages from the developer to the end user, and how this adoption can be used as a model for other package managers looking to improve software distribution and update security. packagingcon-2021-12034-defending-against-attacks-on-package-managers Supply Chain Security Joshua LockMarina Moore en The Update Framework (TUF) is a CNCF graduated project that provides a specification and reference implementation for securing software update systems and other types of content repository. It is used in practice by a diverse range of applications; from single application updaters, through operating systems to automotive firmware update systems and package managers like pip and Composer (for Drupal). TUF was designed to specifically counter previous attacks on software update systems and to create a simple, compromise-resilient framework that will make supply chain attacks on software update systems much harder. This talk will be valuable to maintainers of package managers that support software updates. It will provide information about attacks that package managers may be vulnerable to, as well as tools to prevent these attacks. The audience will come away with a practical understanding of TUF that they can bring back to their projects to improve security either by implementing TUF directly, or by applying some of the principles to make modular improvements to security. false https://pretalx.com/packagingcon-2021/talk/YRTFG9/ https://pretalx.com/packagingcon-2021/talk/YRTFG9/feedback/ Room 3 Talk 2021-11-09T17:25:00+00:00 17:25 00:20 Software package managers have become a vital part of the modern software development process. They allow developers to easily adopt third-party software and streamline the development process. However, bad actors today reportedly leverage highly sophisticated techniques such as typo-squattng and social engineering to “supply” purposefully harmful code (malware) and carry out software supply chain attacks. For example, eslint-scope, a NPM package with millions of weekly downloads, was compromised to steal credentials from developers. We are building a large-scale automated vetting infrastructure to analyze millions of published software packages and provide actionable insights into their composition and security posture. In this presentation, we will cover the technical details of our system and introduce a free tool for developers to detect accidental installation of “risky” packages and mitigate software supply chain attacks. We have already detected a number of abandoned, typo-squatting, and malicious packages. We will present our findings, highlight different types of attacks and measures that developers can take to thwart such attacks. With our work, we hope to enhance productivity of the developer community by exposing undesired behavior in untrusted third-party code, maintaining developer trust and reputation, and enforcing security of package managers. packagingcon-2021-12173-mitigating-open-source-software-supply-chain-attacks-with-ossibot Supply Chain Security Ashish BijlaniAjinkya Rajput en false https://pretalx.com/packagingcon-2021/talk/7L88W8/ https://pretalx.com/packagingcon-2021/talk/7L88W8/feedback/ Room 3 Talk 2021-11-09T17:50:00+00:00 17:50 00:20 Open source software communities rely heavily on user trust. However, typosquatting, watering hole attacks, and developer infrastructure exploits can easily undermine the same honor system that enables easy software package reuse. To better understand trust-based code reuse within language-based ecosystems like npm and Python Package Index (PyPI), IQT Labs recently surveyed 150 software engineers, data scientists, and web developers. Despite high levels of educational attainment, the majority of survey takers agreed with the statement “I wish I knew more about security vulnerabilities associated with code reuse.” When asked who is responsible for keeping code safe, more than half of respondents indicated security is a responsibility individual developers share with package registries. However, this diffusion of responsibility and assumption that package registries have adequate resources to address today's shared code vulnerabilities can lead to developer complacency, particularly since many participants admitted they “do not engage in pre-install code vetting.” In addition to discussing the value of more training, clearer policies, and more robust organizational support, this talk explores the importance of package manager usability. packagingcon-2021-12010-hello-world-a-survey-of-trust-based-code-reuse Supply Chain Security George P. SieniawskiJohn Speed Meyers en * Original survey instrument: https://www.surveymonkey.com/r/codereuse * Summary of the work: https://www.iqt.org/code-reuse-holy-grail-or-poisoned-chalice/ * Data and visualizations: https://www.howdoyou.dev/ true https://pretalx.com/packagingcon-2021/talk/ETZHPY/ https://pretalx.com/packagingcon-2021/talk/ETZHPY/feedback/ Room 3 Talk 2021-11-09T18:15:00+00:00 18:15 00:20 We often use pre-built software binaries and trust that they correspond to the program we want. But nothing assures that these binaries were really built from the program's sources and a set of reasonable build instructions. Common, costly supply chain attacks exploit this to distribute malicious software, which is one reason why most software is delivered through centralized, highly secured providers. Trustix, our reference implementation of a new concept we like to call "build transparency", solves this in an entirely different, decentralized manner. We can accomplish this by leveraging the transparency properties of purely functional package managers such as Nix and coupling this with transparency logs that can be cross compared across multiple independent trust roots. This talk will guide you through the general ideas and concepts underlying this idea and the practical challenges in implementing such as system. packagingcon-2021-12181-trustix-a-new-model-for-trust-in-binary-software-distribution Supply Chain Security adisbladis en false https://pretalx.com/packagingcon-2021/talk/H8FDJL/ https://pretalx.com/packagingcon-2021/talk/H8FDJL/feedback/ Room 3 Talk 2021-11-09T19:00:00+00:00 19:00 00:20 The Python Package Index (PyPI) is one of the oldest software repositories for a language ecosystem and the canonical place to publish Python code. It serves more than 2 billion requests a day, and is almost entirely supported by volunteers and the non-profit Python Software Foundation. In this talk, we'll review some recent supply-chain attacks and how they relate to PyPI specifically. In addition, we'll take a look at some in-progress projects to make PyPI more resilient, secure and sustainable. packagingcon-2021-12160-pypi-supply-chain-security Supply Chain Security Dustin Ingram en false https://pretalx.com/packagingcon-2021/talk/GVV7PU/ https://pretalx.com/packagingcon-2021/talk/GVV7PU/feedback/ Room 3 Talk 2021-11-09T19:25:00+00:00 19:25 00:20 Why everyone should do reproducible builds and how can package managers help in getting there. packagingcon-2021-11965-why-everyone-should-do-reproducible-builds Supply Chain Security /media/packagingcon-2021/submissions/BGXP3D/logo-text_INoA75t.png Bernhard M. Wiedemann en Distributors and users alike are worried these days about supply chain attacks as those on SolarWinds. For FLOSS developers, reproducible-builds is an easy way to let people verify that the published packages indeed correspond to their public sources. This presentation will answer the Why? What? and How? false Slides https://pretalx.com/packagingcon-2021/talk/BGXP3D/ https://pretalx.com/packagingcon-2021/talk/BGXP3D/feedback/ Room 3 Talk 2021-11-09T19:50:00+00:00 19:50 00:20 Golangs module and dependency system addresses more than version management. This talk will explore the lesser known features which support security in the software supply chain. packagingcon-2021-11963-go-mod-s-lesser-known-features-for-supply-chain-security Supply Chain Security Tony Worm en Golang uses the Minimum Version Selection (MVS) to select module versions. This deterministic algorithm has nice properties for reproducible builds and avoids the NP-complete runtime complexity. However, when one digs into the details, they find an array of features and techniques which also support security in the supply chain. The holistic approach from algorithms to tooling demonstrates the experience and expertise that went into designing Go's dependency management system. false https://pretalx.com/packagingcon-2021/talk/X7U9LU/ https://pretalx.com/packagingcon-2021/talk/X7U9LU/feedback/ Room 3 Talk 2021-11-09T20:15:00+00:00 20:15 00:20 Package Managers are an increasingly popular target of attack. Their near-ubiquity in many software ecosystems places developers and end-users at risk while their critical supply chain role makes code execution a frequent consequence of compromise. However with this centralized risk, there is centralized opportunity: Even modest process and policy changes stand to markedly improve each package manager's respective ecosystem. The limited resources available to maintainers should be spent where they can deliver the greatest security benefit. To this end, we present high-value interventions that apply standardized tools and frameworks like Supply-chain Levels for Software Artifacts (SLSA) to the generalized package management domain. packagingcon-2021-12262-securing-the-supply-chain-with-slsa Supply Chain Security /media/packagingcon-2021/submissions/XM3MAB/slsa-dancing-goose-logo_UycXL8P.jpg Matthew Suozzo en It's an old refrain that the security ideal is for your code to run nowhere and do nothing. Therein lies the original sin of software packaging: It helps software run anywhere and do anything. What's worse, most of it is _other people's software_. Fundamentally, package managers facilitate reuse with the aim of making developers more productive. Common abstractions or tools need only be written and packaged a handful of times to serve an entire ecosystem. This incentive structure often leads package managers to prioritize flexibility, stability, and ease-of-use over security and authenticity. But even if these priorities could be inverted, code reuse is simply too valuable and too widespread to give up. If anything, we can expect a relentless increase in the depth and breadth of package dependency graphs. From this somewhat gloomy premise, how do we manage this growing complexity? We posit that the only scalable, generalized option to address these supply chain security concerns is automated dependency graph analysis. Graph analysis can utilize metadata like author identity, source origin, and packaging procedure to track packages' security posture, vulnerability status, etc. in an ecosystem-agnostic fashion. But for this sort of analysis to provide any security value, we need to have trustworthy metadata in standard data formats. Supply-chain Levels for Software Artifacts (SLSA) provides a suitable framework for both standardized data formats and tracking progressive compliance. And instead of rooting trust in elaborate public key infrastructure, we propose bootstrapping it off of existing, durable developer identities. Finally, ecosystem change is never easy, perhaps least of all when it involves new security controls. As such, these building blocks are purposely easy to deploy, adaptable to various ecosystems, and provide sufficient incentive to make implementation worthwhile. false https://pretalx.com/packagingcon-2021/talk/XM3MAB/ https://pretalx.com/packagingcon-2021/talk/XM3MAB/feedback/ Room 4 Talk 2021-11-09T17:00:00+00:00 17:00 00:20 Flatpak-builder is a wrapper around various Flatpak commands to simplify packaging software including, but not limited to, from source. But what if your application is already built as part of CI/CD pipeline, or the host Linux distribution has user namespaces disallowed? Let's have a look at what flatpak-builder actually does and how to flatpak software from scratch. packagingcon-2021-12062-building-flatpak-apps-without-flatpak-builder Metadata & Building things Bartłomiej Piotrowski en false Slides https://pretalx.com/packagingcon-2021/talk/ZWAPSZ/ https://pretalx.com/packagingcon-2021/talk/ZWAPSZ/feedback/ Room 4 Talk 2021-11-09T17:25:00+00:00 17:25 00:20 With hundreds of thousands of open source software (OSS) projects to choose from, OSS is a vital component of almost any codebase. However, with over a thousand unique licenses to comply with, complexity of managing OSS use cannot be overlooked. Identifying and tracking OSS to comply with license requirements adds friction to the development process and can result in product-release delays. At VMware, developers must run a scanner to identify a Bill of Material (BOM) of what OSS is being used. This extra step adds toil and leaves room for error. Some scanners are imprecise, compounding these issues. We solve this problem using Bazel to create an accurate BOM containing OSS and third-party packages during a build. To do this, we made a Bazel aspect that analyzes the dependency graph and collects information about each package from VMware's internal Artifactory. Additionally, it consumes a list of approved and denied OSS from VMware's legal team. By moving OSS validation to build time, OSS decisions are made earlier in the development and review process, making them less costly. packagingcon-2021-12079-streamlining-vmware-s-open-source-licensing-compliance-with-bazel Metadata & Building things Daniel Machlab en Our Bazel aspect outputs two files. First, it creates a BOM yaml file, which includes information on each OSS dependency. Second, it creates a BOM-issues file, containing a subset of OSS dependencies that have been denied for use by the legal team or that are still waiting for approval. A Jenkins server uses the BOM to file legal-review tickets for newly-added OSS. Release managers and developers can use the BOM-issues to identify problems, and the existence of issues can fail a build. We would like to present our work because we hope it will inform the design of general-purpose licensing infrastructure for the Bazel community. false https://pretalx.com/packagingcon-2021/talk/VPV999/ https://pretalx.com/packagingcon-2021/talk/VPV999/feedback/ Room 4 Talk 2021-11-09T17:50:00+00:00 17:50 00:20 Containers and software packages share many traits, but there are also many key attributes lacking in the container management ecosystem that are otherwise present in the package management ecosystem. The popular thinking is that containers do not need package management as those tasks either don’t apply or can be delegated to a higher level orchestrator. The consequence of missing patterns from the packaging community is a less robust and less consistent user experience in distributed cloud compared to what we experience in other domains. This talk will discuss similarities (eg: state management, configuration, and organization of packages into meta-packages) and differences (eg: weak versioning, metadata inclusion, and build determinism) in the container ecosystem compared with familiar package management ecosystems and propose potential improvements to container management inspired by learnings from the package management space. packagingcon-2021-12128-containers-what-s-package-management-got-to-do-with-it Metadata & Building things Nisha Kumar en The container build and distribution ecosystem is the foundation for cloud native applications. Containers are built in the same way that one would configure their desktop to run an application, ie: There isn’t much thought given to the packaging aspects of the container ecosystem such as build repeatability, dependency management, and compatibility solving. These topics are “managed by the orchestrator”, but this is actually implemented with arcane layers of multiple nested orchestrators in control loops, with nesting config and config file generators. This is complicated, indirect, and brittle. Package management patterns exist today and already solve most of the problems containers are facing. The patterns may need to be applied slightly differently of course as containers are a new type of package. As noted in the abstract this talk will discuss key attributes from the state of the art of packaging and package management systems which are present in the containers ecosystem and other missing attributes. Finally, we explore some opportunities present in the gap between the current container build and distribution ecosystem and the state of the art in packaging, and how bridging this gap will result in a more coherent user experience at the orchestration level and beyond. false https://pretalx.com/packagingcon-2021/talk/YZFJDR/ https://pretalx.com/packagingcon-2021/talk/YZFJDR/feedback/ Room 4 Talk 2021-11-09T18:15:00+00:00 18:15 00:20 Helm is the long standing package manager for Kubernetes. Helm packages, called charts, are installed from distributed repositories. In this session you'll learn how Helm came to be, how Helm works, and why it was designed this way. This will include how Helm handles dependencies, how charts are created, signing and verification, and more. packagingcon-2021-12047-how-helm-the-package-manager-for-kubernetes-works Deep Dives Matt Farina en Helm has been the package manager for Kubernetes since near the beginning. Development on Helm began shortly after Kubernetes 1.0 was released. Since then, Helm has grown in use and popularity. Kubernetes is different from an operating system, like Linux, or a programming language. Helm handles package management in a Kubernetes native manner while building on lessons learned from other package managers. In this session you will learn a little about how Kubernetes works, how Helm leverages Kubernetes, how Helm handles typical package management features, and where Helm can continue to improve. false https://pretalx.com/packagingcon-2021/talk/9LSDES/ https://pretalx.com/packagingcon-2021/talk/9LSDES/feedback/ Room 4 Talk 2021-11-09T19:00:00+00:00 19:00 00:20 Cloud Native Buildpacks makes building container images a breeze. It comes with out-of-the-box support for rebasing, reproducibility, multiple entrypoints and more! In this talk we’ll uncover the magic that the lifecycle - the binary at the heart of CNB - uses to convert source code into OCI images. packagingcon-2021-12001-unraveling-the-magic-behind-buildpacks Metadata & Building things Sambhav KothariNatalie Arellano en Cloud Native Buildpacks transform your application source code into runnable images - without Dockerfiles. ## Why is this helpful? - It allows application developers to focus on what they’re building, and not on how to support it in production - It gives operators precise control over what build inputs are permitted and how builds are executed - Lastly operations like rebase that allow mass-patching the base image can have dramatic consequences for large-scale reactions to OS vulnerabilities ## Takeaways In this talk we will explore how lifecycle - the binary at the heart of buildpacks - makes all this possible. We will be going through the 5 different stages that the lifecycle executes - detect, analyze, restore, build and finally export and how it stitches up the final OCI image. false https://pretalx.com/packagingcon-2021/talk/GUBVGF/ https://pretalx.com/packagingcon-2021/talk/GUBVGF/feedback/ Room 4 Talk 2021-11-09T19:25:00+00:00 19:25 00:20 Unikernels are a new way of deploying individual applications as virtual machines in the cloud that can run linux applications faster and safer than linux. Since unikernels are deployed as virtual machines, packaging allows end-users to run common software without compiling it themselves in a cross-platform and cross-architecture way. packagingcon-2021-11961-creating-open-source-unikernel-packages Metadata & Building things Ian Eyberg en Participants will learn about how unikernel packages are made for the open-source Nanos unikernel and OPS tooling. They'll also learn basic unikernel concepts and how packages provide a base for end-users to run common software such as language interpreters and databases. Users will learn how to convert docker containers into unikernel packages quickly and easily. We'll show how to debug when package creation goes wrong and show how unikernel packaging stops many software supply chain attacks. false https://pretalx.com/packagingcon-2021/talk/SHNDF3/ https://pretalx.com/packagingcon-2021/talk/SHNDF3/feedback/ Room 4 Talk 2021-11-09T19:50:00+00:00 19:50 00:20 Three years of community-oriented software bill of materials (SBOM) work under NTIA has lead to (among other things): * Framing of a model, architecture, and requirements for SBOMs, data, and processes * Formats that satisfy the framing constraints: SPDX, CycloneDX, SWID To scale, and really to function at all, SBOM production needs to happen during software development phases such as build, *packaging*, and deployment. We informally reviewed a handful of package management systems to look for commonality, differences, and alignment with the NTIA SBOM effort. One clearly identified SBOM use case, vulnerability management, stands to benefit from more and higher quality SBOM and inventory information. What kinds of data does vulnerability management need from SBOM? To what extent do package management systems provide this data? What are the common elements that package management systems already provide? packagingcon-2021-12042-sbom-packaging-and-vulnerabilities Supply Chain Security Kate StewartArt Manion en We are looking for informed input and potential collaboration to help establish: 1. How widely does package management metadata vary across ecosystems? 2. How well does available metadata during packaging meet SBOM and vulnerability management needs? false https://pretalx.com/packagingcon-2021/talk/TVTEDP/ https://pretalx.com/packagingcon-2021/talk/TVTEDP/feedback/ Room 4 Talk 2021-11-09T20:15:00+00:00 20:15 00:20 The LLVM project encompasses the LLVM core libraries, clang, lld, lldb, compiler-rt, flang and many other projects that gravitates around the use of theLLVM compiler infrastructure. As a whole, they aim at providing a complete tool chain, and its modular structure as led to the developement of many third-party packages such as the Zig language or the Source Trail code explorer. Packaging LLVM leads to numerous choices, from configuration to build, test, installation and granularity point of view. This talk discusses some of these choices in the context of the Fedora distribution. packagingcon-2021-12108-packaging-llvm Metadata & Building things Serge « sans » Paille en This talk is very likely to discuss the following topics : - Why and how do we ship independent packages while LLVM uses a mono repo upstream - Which versioning policy when upstream ships every 6 months? - Building with Clang or with GCC? - It's a compiler... Any bootstraping issue? - How do we cope with the lack of ABI stability between majr version upstream - What is the impact of shipping core components as shared libraries while upstream defaults to static libraries - Any tip to deal with long build time, swapping during linkage and/or package size for large C++ libraries like libLLVM.so? - How do we ensure decent (unit / integration) testing? false https://pretalx.com/packagingcon-2021/talk/JZVZWN/ https://pretalx.com/packagingcon-2021/talk/JZVZWN/feedback/ Plenaries Lightning Talk 2021-11-10T21:15:00+00:00 21:15 00:02 In order to run Robotic Process Automation (RPA) robots, we need Python environments, but we need to set them up cross-platform, isolated, repeatable and fast. RCC enables us to do this based on the conda.yaml config file and by leveraging micromamba, conda-forge and pip. packagingcon-2021-13671-rcc-open-source-tool-to-setup-cache-and-maintain-isolated-and-repeatable-environments-for-the-end-users-and-cloud Kari Harju en Video presentation: * https://drive.google.com/file/d/1snLdbJzB-uyyMjCoCOhXhIzo0nH3NabO/view?usp=sharing Links: * https://github.com/robocorp/rcc#readme * https://github.com/robocorp/rcc/blob/master/docs/environment-caching.md false https://pretalx.com/packagingcon-2021/talk/LHFKYW/ https://pretalx.com/packagingcon-2021/talk/LHFKYW/feedback/ Plenaries Lightning Talk 2021-11-10T21:20:00+00:00 21:20 00:02 Lighting talk about the mass rebuild in Copr. packagingcon-2021-13666-how-we-rebuilt-all-160k-gems-from-rubygems-org-as-rpm-packages Miroslav Suchy en false https://pretalx.com/packagingcon-2021/talk/EWEWNJ/ https://pretalx.com/packagingcon-2021/talk/EWEWNJ/feedback/ Plenaries Lightning Talk 2021-11-10T21:25:00+00:00 21:25 00:02 This lightning talk will offer an answer to the question: What is a Package Manager? The talk will feature slides that were removed from my Lxroot presentation due to time constraints. packagingcon-2021-13674-what-is-a-package-manager Parke Bostrom en false https://pretalx.com/packagingcon-2021/talk/TYEFV3/ https://pretalx.com/packagingcon-2021/talk/TYEFV3/feedback/ Room I Talk 2021-11-10T16:00:00+00:00 16:00 00:45 We’ve managed to bring all of you together from different package manager communities, but can we also bring the package managers you work on together? Is there room for one package manager to rule them all, or will package management always be a very domain-centric activity? If it does, is that good or bad? packagingcon-2021-13658-panel-package-manager-convergence-what-stands-in-the-way Wolf VollprechtTodd GamblinLudovic CourtèsJoshua LockAndrew Nesbitt en We’ve brought together a panel of experts to hash this out! Please welcome: - Wolf Vollprecht (QuantStack, Mamba project, Condaforge) - Joshua Lock (VMWare, The Update Framework) - Ludovic Courtès (Inria, Guix project) - Andrew Nesbitt (Octobox, manifest.fm) Todd Gamblin (LLNL, Spack project) will moderate. Each panelist will briefly give us their take on the following charge questions, followed by a lively discussion and questions from the audience. - Do you think there are too many package managers? Why or why not? - What parts of packaging could eventually be automated or replaced by infrastructure? - What parts of package managers do you think could be shared as common components? What would it take to make that happen? - If not through common components, what other ways could the many different packaging communities come together in the future? - What do we need to do to get people outside the packaging community to better understand these challenges? false https://pretalx.com/packagingcon-2021/talk/TFA8SB/ https://pretalx.com/packagingcon-2021/talk/TFA8SB/feedback/ Room I Talk 2021-11-10T17:00:00+00:00 17:00 00:20 Ray Donnelly liked to say that software collections were defined by "islands of compatibility" - sets of software where the API and ABI requirements line up. Each package ecosystem defines their island differently, and each approach has advantages and disadvantages. This talk will compare the approaches of operating system maintainers, the greater conda ecosystem, and the somewhat ad-hoc status quo of the R world, in the hopes of making implicit assumptions and consequences explicit. packagingcon-2021-12209-islands-of-compatibility ABI & Static Analysis Michael Sarahan en false https://pretalx.com/packagingcon-2021/talk/HWRHTX/ https://pretalx.com/packagingcon-2021/talk/HWRHTX/feedback/ Room I Talk 2021-11-10T17:25:00+00:00 17:25 00:20 Programs crash. And when they do, they dump core, and we want to tell the user which package, including the version, caused the failure. This talk describes a compact JSON-based format that is embedded directly in the binaries as an ELF note. By embedding the this information directly in the binary object, package information is immediately available from a core dump, independently of any external packaging metadata. This is a cross-distro collaboration, with the eventual goal of having the same metadata automatically added by all distributions. packagingcon-2021-11951-package-information-on-elf-objects ABI & Static Analysis Zbigniew Jędrzejewski-SzmekLuca Boccassi en The general idea is to add a terse JSON note as `.note.package` ELF note. This note identifies who and where built the binary: the distro or vendor, distro version, package version, architecture, etc. This is useful when programs compiled for different distributions are mixed (e.g. Debian container running on Fedora), when non-distribution programs are used with a distro (e.g. a private program), when distribution metadata has been stripped (e.g. an initrd image), or when only offline access is possible. This metadata can cooexist with the existing `.note.gnu.build-id` ELF notes. Those are added by most distributions, but they only provide a build-id hash. To resolve this hash to an actual package name and version, some additional query is required. Links: - https://systemd.io/COREDUMP_PACKAGE_METADATA/ - https://github.com/systemd/package-notes - https://fedoraproject.org/wiki/Changes/Package_information_on_ELF_objects false https://pretalx.com/packagingcon-2021/talk/AVRPRC/ https://pretalx.com/packagingcon-2021/talk/AVRPRC/feedback/ Room I Talk 2021-11-10T17:50:00+00:00 17:50 00:20 Updating to a new version of a third-party library is traditionally not a trivial task. Github's dependabot, Renovate, and similar services automatically create a new branch with the latest version of a library dependency and then execute project tests to detect any breaking changes. While such services are gaining a lot of traction, no study looks into whether test suites of average Github Projects have sufficient coverage and are adequate to detect incompatible library changes. To better understand the state of test coverage and effectiveness of project test suites for detecting incompatible library changes, I will, in this talk, present a study comprising 262 Java projects on Github. By artificially injecting faulty changes in library dependencies, we identify that test suites on average have coverage of 58% of their direct dependencies and 20% of their transitive dependencies. The average test suite effectively detects 47% of faulty updates in direct dependencies and 35% in transitive dependencies. Based on our findings, I will explain a set of recommendations for both developers and toolmakers that could potentially improve the reliability and expectations of automated dependency updating. packagingcon-2021-12094-are-project-tests-enough-for-automated-dependency-updates-a-case-study-of-262-java-projects-on-github ABI & Static Analysis Joseph Hejderup en false https://pretalx.com/packagingcon-2021/talk/M9YCL3/ https://pretalx.com/packagingcon-2021/talk/M9YCL3/feedback/ Room I Talk 2021-11-10T18:20:00+00:00 18:20 00:20 `npm audit`, `cargo audit`, `dependabot`, and similar analyzers have one thing in common: they provide feedback by only analyzing project manifests. I have one big problem with this: we are generalizing how projects use dependencies through metadata analysis! Without looking into how projects "actually" use dependencies, we deprive developers of insightful feedback that could save development time and effort. In this talk, I will discuss the differences and similarities between metadata-level versus code-level (i.e., static analysis) dependency analyses. Specifically, I will explain scenarios that are sufficient to use metadata analysis and when it is not. Moreover, I will also discuss the general applicability and challenges of adopting static analysis in dependency analyzers. The talk is based on my research paper: "Präzi: From Package-based to Call-based Dependency Networks" You can find the paper here: https://arxiv.org/abs/2101.09563 packagingcon-2021-12093-the-promises-and-perils-of-adopting-static-analysis-in-dependency-analyzers ABI & Static Analysis Joseph Hejderup en false https://pretalx.com/packagingcon-2021/talk/ASAJQY/ https://pretalx.com/packagingcon-2021/talk/ASAJQY/feedback/ Room I Talk 2021-11-10T18:45:00+00:00 18:45 00:20 Defining dependency relationships is a fraught but integral part of the packaging process. Incorrect dependency definitions can have catastrophic consequences for users and the broader ecosystem. One of the reasons that specifying dependencies is so difficult is because version numbers are very loosely related to the actual property developers care about, the API and ABI. Software doesn’t break if any API changed in a dependency, they only break if the API it relied on changed. Most version number do not capture this, providing a global view of a local problem. To address this, the symbol-management project has begun to catalog as many symbols as possible in the python ecosystem. While this was initially aimed at enhancing conda-forge’s dependency metadata, the implications of the database are much greater. In addition to providing version constraint suggestions on dependencies, the project also enables the creation of version numbers based on changes in the project’s symbols and determination of if a code-base is compatible with a given environment. In this talk I’ll discuss the structure and motivations of the symbol-management project, some examples of how to use the project, and the future of the project. packagingcon-2021-12156-will-the-real-slugify-please-stand-up-adventures-in-api-mapping-and-dependency-discovery ABI & Static Analysis CJ Wright en false https://pretalx.com/packagingcon-2021/talk/P3983F/ https://pretalx.com/packagingcon-2021/talk/P3983F/feedback/ Room I Talk 2021-11-10T19:30:00+00:00 19:30 00:20 [`BinaryBuilder.jl`](https://binarybuilder.org/) is a framework that allows you to compile binaries for an ever-growing set of platforms (16 currently): Linux, FreeBSD, macOS and Windows on various architectures. While `BinaryBuilder.jl` is mainly employed to build libraries and programs used in packages for the [Julia programming language](https://julialang.org/), it is completely general and anyone can install and use on their system the binaries it produces. packagingcon-2021-12052-binarybuilder-jl-using-julia-s-pkg-to-deliver-binary-libraries ABI & Static Analysis /media/packagingcon-2021/submissions/SDEMUJ/build_all_the_things_HUxqyF9_tyfXVfz.jpg Elliot SabaMosè Giordano en The Julia programming languages promises to solve the so-called "two-language problem", so that users don't need to rewrite code in other languages to achieve better performance. However, we don't live in a Julia-only world, and many high-quality libraries that we don't want to miss out have been already written in other languages. We then must jump through hoops to ensure that code written in other languages is easily accessible from Julia. `BinaryBuilder.jl` allows you to take software written in compiled languages such as C, C++, Fortran, Go or Rust, and build precompiled binaries for a plethora of different platforms that can be used from Julia packages, but not only. Automatically generated thin wrappers, called JJLs, allow users to seamlessly install binaries just like regular Julia packages, and the other Julia packages can depend on JLLs. Additionally, Julia's package manger records JLLs in the manifest file capturing the content of an environment for reproducibility, so that not only does it record Julia packages, but also binary libraries in other languages. `BinaryBuilder.jl` cross-compiles binaries for all target platforms in a single sandbox environment, using consistent toolchains. Built binaries are relocatable and packaged into reproducible tarballs. `BinaryBuilder.jl` also takes into account different ABIs ([libstdc++ dual ABI](https://gcc.gnu.org/onlinedocs/libstdc++/manual/using_dual_abi.html) and different libgfortran versions), and all binaries are audited to automatically fix common errors (e.g., check correct ISA and OS ABI for the target platform, set rpaths, etc...). In this talk we will describe the architecture of `BinaryBuilder.jl`, and how it piggybacks Julia's package manager and the Artifacts system to safely deliver the binaries to the end users. false https://pretalx.com/packagingcon-2021/talk/SDEMUJ/ https://pretalx.com/packagingcon-2021/talk/SDEMUJ/feedback/ Room I Talk 2021-11-10T19:55:00+00:00 19:55 00:20 Semantic Versioning (`MAJOR.MINOR.PATCH`) is a common approach to versioning libraries that separates changes into fixes (`PATCH`), additions (`MINOR`), and breakages (`MAJOR`). Though simple, SemVer has two primary limitations that can make it difficult for developers to work with: 1. User-facing changes, such as new features or redesigns, are not separated from API breakages. Therefore, the compatibility between versions is harder for maintainers to understand as the impact of MAJOR updates can vary significantly (ex. Python `1->2` vs `2->3`). In consequence, some projects now use year-based versioning or 'ZeroVer' (where `MAJOR` is always `0`), thus avoiding the question of API compatibility entirely. 2. API breakages are always represented by the `MAJOR` version and do not take into account different types of breakages, such as source vs binary compatibility. Additionally, tooling can be used to repair many common types of breakages (such as renaming) which do not have significant impact on how the library is used. The purpose of this talk is to raise awareness of these limitations, demonstrate the use cases for having multiple levels of API versioning, and propose alternative versioning methods that can incorporate different types of API breakages. packagingcon-2021-12183-versioning-for-user-facing-changes-vs-api-breakages ABI & Static Analysis /media/packagingcon-2021/submissions/JHYQRP/Versioning_for_User-Facing_Changes_vs_API_Breakages_cxKRDBj.png Blake Anderson en - [YouTube Recording](https://www.youtube.com/watch?v=otomn9veySQ) - [Slide Deck](https://docs.google.com/presentation/d/1CO08MZPEhREuLODZ-rr_2aGAYEls6Ti3pkulNn8cFjM) (sources & additional resources in speaker notes) Semantic Versioning (`MAJOR.MINOR.PATCH`) is a common approach to versioning libraries that separates changes into fixes (`PATCH`), additions (`MINOR`), and breakages (`MAJOR`). Though simple, SemVer has two primary limitations that can make it difficult for developers to work with. The first limitation is that user-facing changes (such as new features or redesigns) are not separated from API breakages, which itself causes two issues: 1. Compatibility between versions is harder for maintainers to understand as the impact of MAJOR updates can vary significantly (ex. Python `1->2` vs `2->3`). Some release may cause breaking changes even though the overall library works the same, while others may maintain backwards compatibility but offer new (ideally better) features (ex. Java 8, which added lambdas introducing new options for API design). 2. Developers hesitate to make the `1.0.0` release (and other major releases) for reasons related to the above as well as the effort involved in getting key downstream dependencies to update to avoid compatibility issues. Some projects now use year-based versioning or 'ZeroVer' (where `MAJOR` is always `0`), thus avoiding the question of API compatibility entirely . Second, not all API breakages are the same. The most common example of this is binary breakages, where compiled output fails to with new versions but the source itself remains compatible. Even at the source level, some types of breakages like renaming are 'simple' and can be automatically repaired with the appropriate tooling, while others can require major refactors. There are also breakages specific to the type of application - as a surprising example, Minecraft Forge (a Minecraft API for client-side mods) encourage mods to use separate version for mod compatibility as certain types of changes may break player's worlds (effectively, data versioning). There are a few potential solutions to this, and unfortunately all of them make versions just a bit more complicated. The most straightforward one is to maintain two versions - a true semantic version for the project, and an API compatibility version: - Project version: `PROJECT.FEATURE.PATCH`, which represents the high-level changes to the project as it evolves. - API compatibility version: `PROJECT.SOURCE.BINARY`, which represents the API breakages to the project. `SOURCE` and `BINARY` compatibilities work as expected, but there is an additional `PROJECT` level for changes that have a large impact to how the API can be used. This is currently heuristic, but the inclusion of automated migrations may be able to formalize the idea of 'minor' breaking changes versus 'major' ones to provide strict validation. In summary, software version needs to better account for the difference between user-facing changes and API breakages, as well as account for different types of breakages. One potential solution is to maintain a separate version strictly for API compatibility, coined `PROJECT.SOURCE.BINARY`, to help maintainers understand the potential impact of updating versions. false Slide Deck (see Google Slides link in description for sources & additional readings in speaker notes) https://pretalx.com/packagingcon-2021/talk/JHYQRP/ https://pretalx.com/packagingcon-2021/talk/JHYQRP/feedback/ Room I Talk 2021-11-10T20:15:00+00:00 20:15 00:20 Frequently, reusable packages for major programming languages and operating systems are available in public package repositories where they are developed and evolved together within the same environment. Developers rely on package management tools to automate deployments, specifying which package releases satisfy the needs of their applications. However, these specifications may lead to deploying package releases that are outdated or undesirable because they do not include bug fixes, security fixes, or new functionality. In contrast, automatically updating to a more recent release may introduce incompatibility issues. Moreover, while this delicate problem is important at the level of individual packages, it becomes even more relevant at the level of large distributions of software packages where packages depend, directly or indirectly, on a large number of other packages. The goal of this presentation is to show how to capture this delicate balance between the need of updating to the ideal release and the risk of having breaking changes by presenting the measurement of technical lag, a concept that quantifies to which extent a deployed collection of packages is outdated with respect to the ideal deployment. Then, we empirically analyze its evolution in npm. packagingcon-2021-12224-quantifying-outdatedness-using-the-technical-lag-measurement Metadata & Building things Ahmed Zerouali en false https://pretalx.com/packagingcon-2021/talk/TCRVXX/ https://pretalx.com/packagingcon-2021/talk/TCRVXX/feedback/ Room I Talk 2021-11-10T20:40:00+00:00 20:40 00:20 Semantic versioning (semver) is a commonly accepted open source practice, used by many package management systems to inform whether new package releases introduce possibly backward incompatible changes. Maintainers depending on such packages can use this practice to reduce the risk of breaking changes in their own packages by specifying version constraints on their dependencies. Depending on the amount of control a package maintainer desires to assert over her package dependencies, these constraints can range from very permissive to very restrictive. We empirically compared the evolution of semver compliance in four package management systems: Cargo, npm, Packagist and Rubygems. We discuss to what extent ecosystem-specific characteristics influence the degree of semver compliance, and we suggest to develop tools adopting the wisdom of the crowds to help package maintainers decide which type of version constraints they should impose on their dependencies. We also studied to which extent the packages distributed by these package managers are still using a 0.y.z release, suggesting less stable and immature packages. We explore the effect of such "major zero" packages on semantic versioning adoption. Our findings shed insight in some important differences between package managers with respect to package versioning policies. packagingcon-2021-11843-comparing-semantic-versioning-practices-in-cargo-npm-packagist-and-rubygems Metadata & Building things Tom Mens en The presenter is directing the Software Engineering Lab of the University of Mons in Belgium. The presented results have been published in IEEE Transactions on Software Engineering (https://doi.org/10.1109/TSE.2019.2918315) and Elsevier Science of Computer Programming (https://doi.org/10.1016/j.scico.2021.102656). Slideshare: https://www.slideshare.net/tommens/comparing-semantic-versioning-practices-in-cargo-npm-packagist-and-rubygems Google Scholar profile: https://scholar.google.com/citations?user=5RJe8dsAAAAJ&sortby=pubdate LinkedIn profile: https://www.linkedin.com/in/tommens Twitter handle: @tom_mens false Slides (in pdf format) https://pretalx.com/packagingcon-2021/talk/K3LBCL/ https://pretalx.com/packagingcon-2021/talk/K3LBCL/feedback/ Room 2 Talk 2021-11-10T17:00:00+00:00 17:00 00:20 Package managers are so old that one may wonder why we are here discussing recent tools in this area. What are we trying to achieve that existing tools failed to provide? And why-oh-why does so much energy go into sidestepping package managers through “application bundles” à la Docker? In this talk, I’ll present the grail that GNU Guix is after, taking examples from core features and key packaging practices. You may recognize bits from other projects: the rigor of Debian, the functional paradigm of Nix, the flexibility of Spack. You’ll also see salient differences: Guix tries to go as far as possible in each of these directions while remaining pragmatic. There’s a fine line between pragmatism and deception that Guix tries not to cross. I’ll explain what the project’s “red lines” are and why we think users and implementors should care. I’ll reflect on how we can collectively shape a brighter future for software deployment. packagingcon-2021-11938-the-packaging-grail Deep Dives /media/packagingcon-2021/submissions/R7EG83/guix_AjQAgVI.png Ludovic Courtès en I envision this as a bit of a "soul searching" kind of talk, with the goal of fostering discussion among implementors about what common denominator our tools and distributions should aim for, even when they initially have different goals. false slides https://pretalx.com/packagingcon-2021/talk/R7EG83/ https://pretalx.com/packagingcon-2021/talk/R7EG83/feedback/ Room 2 Talk 2021-11-10T17:25:00+00:00 17:25 00:20 Nix, the package manager for the distribution NixOS, is a package manager built on top of functional programming principles. In this talk I'll discuss how they get close to what I'd consider perfect and what future improvements on the concept should learn from Nix and NixOS. packagingcon-2021-11964-how-nix-and-nixos-get-so-close-to-perfect Deep Dives Xe en false https://pretalx.com/packagingcon-2021/talk/AFCUSQ/ https://pretalx.com/packagingcon-2021/talk/AFCUSQ/feedback/ Room 2 Talk 2021-11-10T17:50:00+00:00 17:50 00:20 This talk introduces conda-forge (a community led collection of recipes for Windows, macOS and Linux), the mamba package manager which works cross-platform and independent of any language and the parts that make it up (libsolv and librepo). Furthermore, we will demonstrate how libmamba can be used to create bindings to mamba or specialized package managers, for example for plugin management in applications. packagingcon-2021-12517-conda-forge-lib-mamba-libsolv-universal-and-reusable-parts Deep Dives Wolf Vollprecht en conda-forge is a community led collection of build-recipes for all major operating systems (Windows, macOS and Linux). It's a hybrid in a space between a classic "Linux"-distribution and PyPI/NPM style DIY distribution. It moves fast and packages a lot of bleeding edge scientific software. conda-forge is fully controlled by repositories on GitHub with humans and bots sharing the work of keeping packages up to date. Mamba is a package manager that works natively with conda packages. It is written in C++ and runs on Windows, macOS and Linux. It also comes with a library (libmamba) and Python bindings (libmambapy) that can be utilized by other applications to manage software or software plugins. Furthermore, recently a new library (currently dubbed powerloader) has been developed that can down- and upload from package repositories. It supports parallel downloads, mirrors and automatic mirror selection as well as OCI registries and S3 storage. It is modeled after librepo but implemented in a more cross-platform compatible way. This talk will introduce all these tools and show opportunities how they can be re-used for other applications. false https://pretalx.com/packagingcon-2021/talk/BVMPCR/ https://pretalx.com/packagingcon-2021/talk/BVMPCR/feedback/ Room 2 Talk 2021-11-10T18:20:00+00:00 18:20 00:20 Fortran is the oldest programming language still in use today, targeting high-performance scientific and engineering applications. Traditionally, Fortran software has used build systems that are not portable or are difficult to use or extend. This has presented a significant barrier to entry for users, and has made it difficult to use libraries as dependencies, or distribute your own library for use in other projects. Fortran Package Manager (fpm) is a new language-specific package manager and build system. The key goals are to improve the user experience and nurture the growth of a rich ecosystem of Fortran libraries. Fpm assumes sane defaults so that most users can enjoy a zero-configuration experience, while providing options to customize behavior. Fpm can scaffold a new Fortran project, fetch and build remote dependencies, and run tests and project executables. It supports multiple compilers, runs on all major operating systems and can bootstrap itself. While new and rapidly developing, it is already used as a build system for large projects and has been met with an overwhelming response from the Fortran community. We want to discuss technical challenges that are specific to building Fortran projects and further next steps. packagingcon-2021-12258-fortran-package-manager-toward-a-rich-ecosystem-of-fortran-packages Deep Dives Sebastian Ehlert en false https://pretalx.com/packagingcon-2021/talk/8MFYFW/ https://pretalx.com/packagingcon-2021/talk/8MFYFW/feedback/ Room 2 Talk 2021-11-10T19:30:00+00:00 19:30 00:20 OPAM is the de facto standard package manager for the OCaml programming language. As a frequent contributor to its repository, I present an overview of its evolution, features, and recent ecosystem projects such as automated lower bounds checking, as well as my own experience with it. packagingcon-2021-11981-living-with-opam Deep Dives Daniil Baturin en OCaml was relatively late to receive a language-specific package manager and OPAM brought a big change to the ecosystem. The fact that it allows managing compiler installations together with packages and that OPAM can be installed _before_ the compiler was a rather novel feature at the time. The talk is focused on the following topics: * Features and packaging workflows. * Positive and negative effects of centralization (with specific examples) * Recent effort on automated lower bounds checking * Evolution, transition to OPAM 2.0 and how it was managed * Ecosystem challenges (Windows support, cross-compilation...) * My personal experience with OPAM and the ecosystem as a long-term contributor false https://pretalx.com/packagingcon-2021/talk/VUTGK3/ https://pretalx.com/packagingcon-2021/talk/VUTGK3/feedback/ Room 2 Talk 2021-11-10T19:55:00+00:00 19:55 00:20 Linux package managers are too slow; how could we make things better? packagingcon-2021-12068-distri-researching-fast-linux-package-management Deep Dives Michael Stapelberg en In my work on distri, I show that package managers can fill almost any line rate (1 Gbps, 10 Gbps, 100 Gbps) effortlessly with the right architecture. In this talk, I will explain the key ideas in distri’s architecture, and what impact they have on the resulting system. For example, packages are distributed as SquashFS images in distri, which makes their contents immutable and the overall system harder to break. false https://pretalx.com/packagingcon-2021/talk/J3CHVC/ https://pretalx.com/packagingcon-2021/talk/J3CHVC/feedback/ Room 2 Talk 2021-11-10T20:15:00+00:00 20:15 00:20 The Haiku operating system, which is a modern, open source re-implementation of BeOS from the 1990's, has an interesting software packaging system. Much like Debian's .deb or RedHat's .rpm files, Haiku's .hpkg files include the files, description of the software, and dependencies. Like it's Linux cousins, it also ensures that the dependencies are met, installing the dependencies if not already installed and available in the repository. What sets Haiku's package manager apart is two things: Each file in the package is mounted as a read-only file into the file system, which ensures security; and the boot manager is aware of the state of the packing system, allowing the user to reboot and start the operating system as it was in a prior state. Since each file is mounted from the package into the file system, it cannot be changed, either by the user (intentionally, or accidentally), or by a mis-behaving application. The only way to change the file is to install a different version, or to uninstall it completely. There is a downside to this though, it does make porting some applications tricky. packagingcon-2021-12054-the-haiku-package-manager Deep Dives Richard Zak en false https://pretalx.com/packagingcon-2021/talk/9V9YQK/ https://pretalx.com/packagingcon-2021/talk/9V9YQK/feedback/ Room 2 Talk 2021-11-10T20:40:00+00:00 20:40 00:20 The TeX environment has grown slowly but steadily to a huge collection of programs, fonts, macros, support packages. Current TeX Live ships about 5Gb in more than 3500 different units. As teTeX stopped to be developed several years ago, TeX Live has taken over as the main TeX distribution in practical all areas, not only on Unix, but also Mac (MacTeX is based on TeX Live) and is also gaining on Windows (where MikTeX is still strong). In this talk we recall shortly the history of TeX Live, its transition from CD/DVD based distribution to net based distribution, and the difficulties one faces when distributing a considerable piece of software to a variety of operating systems and hardware combinations (currently about 15 different arch-os combinations). Topics touched are cross-platform distribution, security, release management etc. Furthermore, we will discuss the topic of re-distributing TeX Live into Linux distributions like Debian, Red Hat. Integrating TeX Live into any distribution is a non-trivial task due to big amount of post installation steps. And although over the last years the quality of packages has improved, we still often get bug reports that stem from incorrect packaging. packagingcon-2021-12031-packaging-tex-live-the-challenge-of-multi-platform-support Metadata & Building things Norbert Preining en false https://pretalx.com/packagingcon-2021/talk/TJ7NJV/ https://pretalx.com/packagingcon-2021/talk/TJ7NJV/feedback/ Room 3 Talk 2021-11-10T17:00:00+00:00 17:00 00:20 Every packaging system has its specific way of doing things, but to an outsider. Python’s seems to have a knack of finding the most non-straightforward and weird solution for every choice. This talk attempts to trace some of the peculiarities to find out the reasoning behind the decisions, and how they stand in the modern packaging landscape. packagingcon-2021-12074-python-packaging-why-don-t-you-just Deep Dives Tzu-ping Chung en Python packaging has a long and winding history. When it startd out, few communities have good packaging offerings, and Python’s was praised for being easy to be picked up and develop with. As time went on, many tried to bring in new ideas from relatively new packaging systems to “modernise” Python packaging, but those suggestions are not always received favourably, and Python’s packaging community is sometimes categorised as overly conservative, or even user-hostile due to this. In most of such situations, however, the main problem is not that Python packaging is unwilling to change, but there are certain design decisions that affect how problems should be approached. While those characteristics may seem alien to newcomers, they are made to accomdate special needs in the Python community, and any changes to the packaging system need to take them into consideration. This sometimes require contributors to be more creative when putting together a new feature, but the Python community is only as valuable as its users, and new progress should avoid throwing away what enable Python to grow to what it is. false Slides (PDF) Slides (Markdown with speaker notes) https://pretalx.com/packagingcon-2021/talk/X8N8ME/ https://pretalx.com/packagingcon-2021/talk/X8N8ME/feedback/ Room 3 Talk 2021-11-10T17:25:00+00:00 17:25 00:20 Python wheel is a beautifully simple format for cross-platform binary distribution. Combining it with the simple repository API, we have the Python Package Index (PyPI) tirelessly serving Pythonistas. PyPI is great as a package index, but in certain ways it is unsuitable for end-user usages: it is subject to multiple supply chain attacks, its centralised nature leads to difficult mirroring while being a single point of failure, and expensive dependency resolution is left for client-side. The interplanetary wheels (IPWHL) are platform-unique, singly-versioned Python binary distributions backed by IPFS. It does not try to replace PyPI but aims to be a downstream wheel supplier in a fashion similar to GNU/Linux distributions, whilst take advantage of a content-addressing peer-to-peer network to provide a reproducible, easy-to-mirror source of packages. packagingcon-2021-12155-ipfs-python-wheels-efficient-secure-and-reproducible-repository Registries Nguyễn Gia PhongHuy Ngo en This talk will first briefly discuss the wheel package format and the current state of PyPI in the Python packaging ecosystem, focusing on a few shortcoming and relevant recent efforts. It will then introduce IPWHL from the motivating philosophy to real-world properties, before showing the current process and a demo usage. As this happens, the upstream-downstream relationship in Python packaging will be analyzed, clarifying the role of each actor in the process. After the talk, the audience will know how (and when) IPWHL can benefit them and the different ways they can directly and indirectly help the project. false Slides https://pretalx.com/packagingcon-2021/talk/RWR89G/ https://pretalx.com/packagingcon-2021/talk/RWR89G/feedback/ Room 3 Talk 2021-11-10T17:50:00+00:00 17:50 00:20 piwheels is a mirror of the Python Package Index, providing binary distributions compiled for the Raspberry Pi's Arm architecture. Package maintainers usually provide wheels compiled for PC/Mac but not for the Arm architecture, so piwheels natively compiles all packages and makes them available to Raspberry Pi users, the regular way, using pip, without any change in behaviour required. Providing pre-compiled binary wheels saves users time and effort, reducing friction to getting started with Python projects on Raspberry Pi. packagingcon-2021-11991-running-a-python-package-index-for-raspberry-pi Registries /media/packagingcon-2021/submissions/EP3ZLT/piwheels_logo_white_Gvyq7O7.jpg Ben Nuttall en piwheels is a mirror of the Python Package Index, providing binary distributions compiled for the Raspberry Pi's Arm architecture. Package maintainers usually provide wheels compiled for PC/Mac but not for the Arm architecture, so piwheels natively compiles all packages and makes them available to Raspberry Pi users, the regular way, using pip, without any change in behaviour required. Providing pre-compiled binary wheels saves users time and effort, reducing friction to getting started with Python projects on Raspberry Pi. We serve over 2 million downloads each month and have saved Raspberry Pi users over 500 years of build time. In this talk I'll explain: - how the build process works - what our infrastructure looks like - how we deal with keeping in sync with the upstream index - building for multiple OS versions and Python versions - compiling Python packages with C/C++/fortran/Golang/Rust/etc - how we make it easier for people to install packages and (apt) system dependencies false https://pretalx.com/packagingcon-2021/talk/EP3ZLT/ https://pretalx.com/packagingcon-2021/talk/EP3ZLT/feedback/ Room 3 Talk 2021-11-10T18:20:00+00:00 18:20 00:20 We believe that sharing and reusing data science code is the future for scaling machine learning across the world because it allows us to work more efficiently. To achieve this grand vision, we had to look at how micro-packaging could be done in Python, the language of choice for most data scientists. Micro-packaging is a widely debated topic in the npm world, and it hasn't taken off in the Python packaging ecosystem. This talk will present the journey that brought us to this point, the challenges we've faced implementing this functionality and the solution we created in Kedro, an open-source Python framework for data science. Whether you're a data practitioner or a software engineer curious to reuse code between projects, you can draw some inspiration from this talk. packagingcon-2021-12085-micro-packaging-reusable-data-science-pipelines-in-python Metadata & Building things Lorena Balan en We have used Kedro to build reusable code stores, similar to how React is used to create design systems. Kedro is an open-source Python framework for creating reproducible, maintainable and modular data science code. While Kedro did lift many barriers, our users found that they still needed a way to easily share code snippets and parts of their data science pipelines between projects. Furthermore, they wanted to consume business logic as *source code*, and possibly extend it, rather than as a *library*. This prompted my team to think about ways to enable a more seamless experience of sharing data science code in ways that didn't confuse our beginner users or force data scientists to take a bunch of software engineering classes to use the feature. Thus micro-packaging was born: packaging (and consuming) submodules using simple CLI commands and a manifest file (pyproject.toml). In this talk, we cover: - Setting the stage - introducing the main pain points we needed to address - How the Kedro solution works - What's under the hood - Reflections & future thinking false https://pretalx.com/packagingcon-2021/talk/9MKDF3/ https://pretalx.com/packagingcon-2021/talk/9MKDF3/feedback/ Room 3 Talk 2021-11-10T18:45:00+00:00 18:45 00:20 The Fedora Python SIG and the Python maintenance team at Red Hat are systems integrators who work at the intersection of two worlds: a cross-platform ecosystem and a platform open to all kinds of software. This talk introduces both Python packaging and RPM, explains why we go through the trouble to repackage Python projects in RPM, and covers some of the issues we're solving. packagingcon-2021-12247-python-in-fedora-integrating-a-language-ecosystem-in-a-distro Metadata & Building things /media/packagingcon-2021/submissions/GYGHXX/fedoralovespython_HqDWOIS.jpg Petr ViktorinMiro Hrončok en Python packaging is evolving quickly over the past few years, with a focus on standards that allow cooperation between different tools. The “traditional” way to package projects – setuptools – is still dominant, but its new aim is to be just one way to do things, so better competitors are able to overtake it. The Python ecosystem is centered on PyPI, the Package Index: a place where anyone can share their project with the world. The usual way to install from PyPI is into *virtual environments*, semi-isolated environments that don't conflict with each other or the base system (when used well). Packages from PyPI are generally easy to install on any platform. RPM is a package format for building a *distro*: collection of packages that's coherent (designed to work together) and complete (with everything from the kernel to the GUI apps). It's used for packaging software written in any language, although it does have a historical bias for C projects and the Autotools *configure-make-install* paradigm, and works much better in ecosystems that care for *distro*-style integration (as opposed to *monorepos*). RPM (and system packages in general) have several advantages over Python (and language ecosystems in general): tighter integration with the system, and common tools for handling packages – installation (so a non-Pythonista can easily get a Python-based tool/dependency), auditing delivery pipelines, system integration and integration testing, and so on. That makes it worth the time to repackage projects from PyPI as RPMs. Python packaging's focus on cooperation between different tools and RPM's focus on combining language ecosystems play together nicely, with opportinities for collaboration and improvements in both ecosystems. Our goal in Fedora is to make repackaging a Python project as RPM as easy as possible. New Python packaging macros and guidelines released in June 2021 reuse metadata from upstream, leaving packagers to focus on system integration: applying necessary changes (hopefully ones that will make it back to the project), running tests (and convincing people that it's nice to have tests pass in environment other than the project's CI), integrating non-Python software, and so on. When bridging the two world, we run into issues with naming, versioning, licenses, testing, documentation, optional dependencies. Some of which are solved easily, some of them need more work and discussion. We hope that the long run, making Fedora and Python work together helps other distros and language ecosystems as well. false https://pretalx.com/packagingcon-2021/talk/GYGHXX/ https://pretalx.com/packagingcon-2021/talk/GYGHXX/feedback/ Room 3 Talk 2021-11-10T19:30:00+00:00 19:30 00:20 Python packaging has changed a lot in the last few years. New tools such as Poetry and Flit allow creating packages without the traditional `setup.py` file, and new standards mean that `pyproject.toml` files are now the linchpin for building and installing Python modules. The wheel package format, which is somewhat older, has also gained a more central role. I’ll explain what has changed, including a brief summary of what motivated these changes. Then I’ll discuss how you can use the new standard interfaces and formats, with a focus on people re-packaging Python packages into other distribution systems such as Conda, Spack, or Linux distribution repositories. I’ll introduce the low-level ‘build’ and ‘installer’ tools, and compare them to the more widely used ‘pip install’. packagingcon-2021-12061-changes-in-python-packaging-what-downstream-packagers-need-to-know Metadata & Building things Thomas Kluyver en false https://pretalx.com/packagingcon-2021/talk/SPQKAL/ https://pretalx.com/packagingcon-2021/talk/SPQKAL/feedback/ Room 3 Talk 2021-11-10T19:55:00+00:00 19:55 00:20 End users think in terms of environments not packages. The core philosophy of conda-store is to serve reproducible conda environments in as many ways as possible to users and services. Conda-store was developed due to a significant need we found in enterprise architectures. There are many ways to serve environments and each plays an important role. Thus conda-store serves the same environment via a filesystem, lockfile, pinned yaml specification, conda pack archive, and docker image. This logic could easily be extended to also support the creation of VM iso's and singularity containers During this talk I will highlight some common problems with environments we have seen while consulting and show how conda-store aims to solve them: - Friction between IT and end users in controlled environments where new packages are needed - Enabling a given notebook developed within jupyterlab to be reproducibly run in workflows reliably for years to come - Helping to removing the need for specially crafted docker containers This talk will be full of demos along with a site that everyone in the talk can try out. packagingcon-2021-12255-serving-and-managing-reproducible-conda-environments-via-conda-store Registries Christopher OstrouchovJaime Rodríguez-Guerra en End users think in terms of environments not packages. The core philosophy of conda-store is to serve identical conda environments in as many ways as possible. Conda Store controls the environment lifecycle: management, builds, and serving of environments. It **manages** conda environments by: - watching specific files or directories for changes in environment filename specifications - provides a REST api for managing environments - provides a command line utility for interacting with conda-store conda-store env [create, list] - provides a web ui to take advantage of many of conda-store's advanced capabilities It **builds** conda specifications in a scalable manner using N workers communicating via Celery to keep track of queued environment builds. It **serves** conda environments via a filesystem, lockfiles, tarballs, and soon a docker registry. Tarballs and docker images can carry a lot of bandwidth which is why conda-store integrates optionally with s3 to actually serve the blobs. Below are highlighted some common problems with environments we have seen while consulting and show how conda-store aims to solve them. ## IT and End User Friction We saw tension between the IT/sysadmins and end users who use the environments that they build. When IT gets a request for a new package in an environment, they need to rebuild the environments and check that the package satisfies their constraints. This process may take several days and at best will not be immediate. While developers need packages in their environments as soon as possible to do interesting new research. This situation often led to a lot of frustration on both sides for good reason. Conda-store aims to address this by allowing users to control a set of environments in their namespace while allowing IT to having all environments under their control. ## Reproducibly Productionizing Environments Another issue we saw was the need to quickly productionize workflows and ensure that they may run for many years to come. Often times developers will experiment with a given environment and create a notebook to run a given workflow. They will want to “submit” this notebook with the given environment and run it on a cron job. The only problem is that this creates a huge burden on IT. How is IT supposed to ensure that the environment that that notebook ran with is preserved indefinitely? Conda-store addresses this by building all environment separately(including updates). There is a unique key that identifies any given environment. Furthermore this environment is available in many different forms: yaml, lockfile, conda tarball, and docker image. The advantage here is that the workflow orchestration framework may run significantly different from the developer environment and we need a way to ensure that environments are the same. - Documentation: https://conda-store.readthedocs.io - Repository: github.com/quansight/conda-store false https://pretalx.com/packagingcon-2021/talk/FGVPWP/ https://pretalx.com/packagingcon-2021/talk/FGVPWP/feedback/ Room 3 Talk 2021-11-10T20:15:00+00:00 20:15 00:20 Both software and documents have dependencies. This talk focuses on managing document dependencies, to reduce both network and computation latency, and to ensure reproducible build (or typesetting) behaviour. Web development has a strong focus on reducing user experienced latency, as does serverless cloud computing. At present human activity and large downloads are required to achieve these goals for TeX documents. To improve matters the speaker has introduced the concept of Portable TeX Documents (PTD). The PTD concept is intended to bring to source documents and the TeX community benefits similar to the benefits Portable Document Format (PDF) brought to Word users and Adobe. The concepts and tools underlying PTD, particularly mounting git as a read-only file system, and the use of git backing stores (alternate object databases) are likely to be useful elsewhere. This is particularly true when most of the variability of a system lies in a small folder of text files (which is the case for TeX's typesetting inputs). packagingcon-2021-12115-tools-for-packaging-and-using-portable-tex-documents Deep Dives Jonathan Fine en Adobe's Portable Document Format (PDF) stores the output of a typesetting process. It includes a structured storage system for storing dependencies such as fonts, graphics, multimedia objects and other resources. The speaker is creating similar tools and standards for the inputs to a typesetting process. It will allow authors and others to collaborate using Portable TeX Documents (PDT). A key technology will be git used with a backing store (ie alternate object database). Each PDT will be a git tree containing all typesetting inputs. Shared resources such as fonts and style files will be placed in the backing store. Being able to mount a git repository as a read-only file system would be very helpful. Use of PTD will greatly reduce the human, bandwidth and storage cost of typesetting a new TeX document in your inbox. It will also greatly increase the sender's confidence that your output will be identical to theirs. The basic idea of PDT is to use git with a backing store. This technology and associated tools will be most helpful when most of the variability of the system lies in a small folder of text files (which is the case for TeX's typesetting inputs). For more information see [video discussion of Portable TeX Documents](https://www.youtube.com/playlist?list=PLw1FZfIX1w7gcm6b4MzRKDIlW7xV4yFnd). * [Slides for my talk](https://www.slideshare.net/jonathanfine/portable-tex-documents-ptd-packagingcon-2021) * [Video of my talk (21min)](https://www.youtube.com/watch?v=oDoQ2G48aqM&t=12039) * Development will take place at https://github.com/arxtex/ptd. false SLIDES for this talk (PDF). https://pretalx.com/packagingcon-2021/talk/XTAJ7Z/ https://pretalx.com/packagingcon-2021/talk/XTAJ7Z/feedback/ Room 3 Talk 2021-11-10T20:40:00+00:00 20:40 00:20 `pip`, Python's package manager, is developed independently from the Python language by a fairly independent team. It has an extensive test suite, with significant complexity and computational requirements. A mix of I/O heavy tests and CPU heavy tests, combined with the wide matrix of supported platforms and Python versions, introduce some interesting challenges when needing to run an overall CI workflow in a reasonable amount of time. This talk goes into the trials and tribulations of getting the CI for pip to run in less than 30 minutes. packagingcon-2021-12299-the-quirks-and-challenges-of-pip-s-test-suite-and-ci Metadata & Building things Pradyun Gedam en false https://pretalx.com/packagingcon-2021/talk/CEVLC3/ https://pretalx.com/packagingcon-2021/talk/CEVLC3/feedback/ Room 4 Talk 2021-11-10T17:00:00+00:00 17:00 00:20 Rust has been around as a language for about 10 years now and a necessary part of distribution packaging for at least the last 4 with Firefox depending on it. In Guix we've been struggling to have a sane way to package rust applications and all their dependencies while trying to keep a handle on visualizing build chains and an ever expanding package set. packagingcon-2021-11990-adventures-in-packaging-rust-programs Metadata & Building things Efraim Flashner en Installing rust programs from the command line using cargo is straightforward, but what happens when distributions get involved and try to package them? With every input coming from the distribution, how can we make sure to track package dependencies when a package depends on the sources of other packages and not their compiled outputs? In Guix we've been packaging all the input crates and using their sources. However, the standard build chain assumes compiled packages as build-time inputs for the next package, and this collapses when we only need the source of the input package. Don't forget the next step! Users of the distributions are supposed to be able to get their development environment from the distribution too! So... how do we give them all these crates in a useful manner? false slides in rst format https://pretalx.com/packagingcon-2021/talk/LVPMSN/ https://pretalx.com/packagingcon-2021/talk/LVPMSN/feedback/ Room 4 Talk 2021-11-10T17:25:00+00:00 17:25 00:20 The Freedesktop SDK began life providing a runtime for the Flatpak app distribution tool. Now Freedesktop SDK generates a variety of base reference systems, including common libraries and utilities for other projects to build on top of. It's not easy to do this reliably, so let's talk about the tools and processes that make this possible. packagingcon-2021-12203-generating-different-base-systems-from-the-same-inputs-in-freedesktop-sdk Sam Thursfield en This talk will focus on how the Freedesktop SDK project: * Controls the entire delivery pipeline to continuously build, test and integrate, keeping the main branch constantly green * Uses BuildStream and plugins to generate Flatpak runtimes, OS images and containers * Guarantees ABI stability * Aims for reproducible builds false https://pretalx.com/packagingcon-2021/talk/AMEDRN/ https://pretalx.com/packagingcon-2021/talk/AMEDRN/feedback/ Room 4 Talk 2021-11-10T17:50:00+00:00 17:50 00:20 As repositories grow in size with packages, the time complexity starts to become O(n*log(n)) to keep the metadata up-to-date, because retaining the history requires re-parsing published packages and those must be available locally. At NVIDIA, the Triforce repository management system handles the release process in O(n). To re-generate the metadata, one or more product release candidates are merged together using OverlayFS, on top of the public repository; this avoids the need for copying hundreds of gigabytes of existing packages, significantly reducing the I/O and storage usage. Another consideration is how long it takes to build the metadata, by default generated from scratch each time. For RPM repositories, createrepo_c has the flag --update which skips over existing packages that have not changed. However for Debian repositories, existing tools such as apt-ftparchive lack such functionality. Comparing the filenames and file sizes is a good enough indicator if the package can be skipped. Parsing dpkg --info, it is possible to form the fields in a deterministic order for each block. From there it is as simple as appending the new metadata to the existing Packages.gz and regenerating the Release file. packagingcon-2021-11984-triforce-repository-management Metadata & Building things Kevin Mittman en Package repositories are a core component of most Linux distributions, providing a mechanism for installing and updating various software components. Package managers require an entry point such as repomd.xml (RPM) or Release (Debian) to discover software available in a repository and using properties stored in a manifest file, like *-primary.xml.gz (RPM) or Packages.gz (Debian) such as version, dependencies, and conflicts to build a transaction to execute. For the CUDA repository, on a per distro / arch basis, everything is stored into one package repository. This is compounded as release candidates for independent products are validated in parallel, where product A may ship before product B, vice versa, or simultaneously. To handle these different scenarios, a train model is implemented in which one or more “passenger” RCs may be released during a given slot. This talk will describe some of the challenges of releasing deliverables on a fast cadence, lessons learned, and finally provide examples of code used in production. false Slides https://pretalx.com/packagingcon-2021/talk/88VTJZ/ https://pretalx.com/packagingcon-2021/talk/88VTJZ/feedback/ Room 4 Talk 2021-11-10T18:20:00+00:00 18:20 00:20 Nix and similar tools (Spack) promise a reproduciblity story for packages (from source or bitwise). Specifically within Nix, several languages have successfully integrated into the ecosystem but some such as Java are oddly absent given their popularity. In a search for how to better integrate Java into a Nix-centric workflow, we go over some current challenges with the fractured Java ecosystem and how the appeal of a federated artifact store has led to sharp edges. packagingcon-2021-11972-challenges-with-java-in-a-hermetic-world Metadata & Building things Farid ZakariaCarlos Maltzahn en - Go over high level basics for what a good integration story into Nix would be for a language - Things that make Java particularly hard: - JARs are ZIP so inherently not reproducible. - Maven artifactories don't enforce a cryptographic hash. Different artifactories can have different content for the same version. - Fractured dependency management software (Gradle, Ivy, Ant, Maven) - Lack of lock file and version ranges (or SNAPSHOT) cause projects to drift over time false https://pretalx.com/packagingcon-2021/talk/NFEWWA/ https://pretalx.com/packagingcon-2021/talk/NFEWWA/feedback/ Room 4 Talk 2021-11-10T18:45:00+00:00 18:45 00:20 Traditionally, building Debian packages is quite complicated. With the "debian" folder that needs to be merged into the source tree with all the various files, the various mechanisms of automagic that you may need to figure out in case it goes sideways, and the hugely over-descriptive yet difficult to understand Debian Policy Manual, it's no surprise that people get it wrong so often! But what if there was a simpler path to making (mostly) conformant Debian package? Enter debbuild, a tool that lets you use the simpler RPM spec file format to build a Debian package. With debbuild, it's possible to easily make portable packaging across all major distributions with very little pain! Come and see how debbuild can help make it easier to ship Linux software the right way! packagingcon-2021-12286-building-debian-packages-the-rpm-way-with-debbuild Metadata & Building things Neal Gompa en This talk introduces debbuild (https://github.com/debbuild/debbuild) and talks about the history and motivation behind the creation of the tool. Additionally, it introduces debbuild-macros (https://github.com/debbuild/debbuild-macros) and walks through how to use both to create RPM spec files that build either RPM packages or Debian packages, with example packages noted. false https://pretalx.com/packagingcon-2021/talk/AHVELM/ https://pretalx.com/packagingcon-2021/talk/AHVELM/feedback/ Room 4 Talk 2021-11-10T19:30:00+00:00 19:30 00:20 Multi-cloud and microservices are making us redefine the meaning of a "package." Modern applications span languages, operating systems, networks, and machines. To deploy a whole service, you need binaries, configuration files, environment variables, host metadata, and services must be connected and secured at runtime. For a developer, it becomes a best practice to save the entire runtime of a service as deployment configuration in version control. Deployment configurations, combined with powerful workload orchestrators, make it easy to guarantee reproducible runtime, but managing these configurations with version control and open-source dependencies starts to resemble package management. For system operators, ensuring that the computing clusters have relevant software packages installed for successful deployments can also be a challenge, as the application package landscape changes rapidly and manual provisioning slows development. To make it easier for developers and operators to embrace DevOps, we built a package manager for deployments running on Nomad, a distributed workload orchestrator. This talk will cover a range of topics related to package management and DevOps workflows, including the best practices we learned while building a package manager to guide users on their journey to multi-cloud. packagingcon-2021-12264-package-management-for-devops Deep Dives Jasmine Dahilig en This talk will cover a range of topics related to package management and DevOps workflows including: - versioning for the cloud - monorepo versus multisource - building a package ecosystem - metaprogramming and dependency injection - DevOps best practices Nomad is a distributed workload orchestrator built at HashiCorp. Nomad is a unique orchestrator because it is workload-agnostic, which means it can run many different types of runtime artifacts on your clusters. It can run any workload that is supported as a Nomad task driver, including Java JARs, QEMU virtual machines, Firecracker microvms, or any raw executable binary. It is also system-aware with Nomad device plugins, allowing you to run your workloads on machines with available devices such as GPUs. false https://pretalx.com/packagingcon-2021/talk/ZHG88M/ https://pretalx.com/packagingcon-2021/talk/ZHG88M/feedback/ Room 4 Talk 2021-11-10T19:55:00+00:00 19:55 00:20 Building proper Debian packages from Dist::Zilla maintained Perl modules, especially from git checkouts without having a Dist::Zilla generated tar ball yet. packagingcon-2021-12026-dh-dist-zilla-from-dist-zilla-s-dist-ini-to-debian-s-deb-in-one-go Metadata & Building things Axel Beckert en [dh-dist-zilla](https://github.com/elmar/dh-dist-zilla) is a [debhelper](https://tracker.debian.org/pkg/debhelper) sequence plugin to be used in the `debian/rules` file of a Debian source package. The intention is to be able to build Debian binary packages (`.deb` files) directly from a Dist::Zilla based Perl package or even git checkout without manually calling "dzil build" to generate the CPAN-wanted files (`META.yml`, `README`, etc.) or even `Makefile.PL` first. It is analogous to using `autoreconf` to generate the configure script. One use case (and the initial motivation for dh-dist-zilla) is internal Perl modules which need to be deployed as Debian binary package and won't be uploaded to CPAN ever. Another use case is to be able to build `.deb` packages directly from git checkouts or tar-balls downloaded from GitHub, i.e. of not-yet released snapshots of a project. Slides at https://noone.org/talks/pkg-perl/dh-dist-zilla-packagingcon2021.html false https://pretalx.com/packagingcon-2021/talk/ZLJFCU/ https://pretalx.com/packagingcon-2021/talk/ZLJFCU/feedback/ Room 4 Talk 2021-11-10T20:15:00+00:00 20:15 00:20 Lxroot is a lightweight software virtualization tool (for Linux). With Lxroot, a non-root user can safely and easily install, run, develop, and test both packages and package managers. Compared with other virtualization tools, Lxroot is safer, smaller, conceptually simpler, and arguably more flexible (within the limits of what is possible as a non-root user). Lxroot allows a non-root user to create chroot-style virtual environments via Linux namespaces. Lxroot simply creates and configures these chroot-namespaces, and then runs programs inside them. All the virtualization work is done directly by the Linux kernel itself, via its namespace capabilities. Lxroot allows the simultaneous use of multiple package managers, both system package managers (such as pacman, apk, xbps, etc.), and non-system package managers (such as pip, npm, Flatpak, conda, mamba, Spack, etc.). Lxroot allows a non-root user, on a single host kernel, to easily mix-and-match packages, userlands, and package-managers from multiple sources, including from multiple different Linux distributions. Due to its simple and flexible nature, Lxroot has a variety of use cases related to the development, testing, and use of packages and package managers. More information here: https://github.com/parke/lxroot packagingcon-2021-11993-lxroot-run-develop-and-test-packages-and-package-managers-in-a-lightweight-virtual-environment Metadata & Building things /media/packagingcon-2021/submissions/PMPUSW/20210929_Lxroot_kBUsrjT.png Parke Bostrom en This talk is an introduction to basic and intermediate use of Lxroot, with a special focus on topics related to packages and package managers. Topics covered: 1. About Lxroot - introduction & goals - related tools - Linux namespaces & performance - lightweight virtualization - limitations 2. Three demos - install a guest userland - demo of non-distro package managers: pip, npm, Spack, Nix, Flatpak - build an Alpine Linux package 3. Discuss - compatibility with distro package managers - Lxroot's command line interface - use cases vis-a-vis building, testing and installing packages - unexplored territory 4. Conclusion More information here: https://github.com/parke/lxroot false Lxroot presentation slides https://pretalx.com/packagingcon-2021/talk/PMPUSW/ https://pretalx.com/packagingcon-2021/talk/PMPUSW/feedback/