2024-07-10 –, REPL (2, main stage)
Almost every Julia user is well acquainted with the package manager, Pkg, and uses it to add, update, and remove packages from their package environments. Behind the scenes Pkg is supported by a global distribution network of Julia-powered package servers that host registries, packages, and artifacts. In this talk we will take a look at how this infrastructure is configured and how it is being used to deliver packages from developers to users.
When adding or updating a package using Pkg the first step is to resolve the package environment. The output of the resolver is the version of the package, and the versions of all dependencies, that should be installed. The next step is to make sure the required packages and artifacts are available on the users machine. Since installation directories of packages (~/.julia/packages) and artifacts (~/.julia/artifacts) are shared between all package environments it is quite likely that some of the required resources are already available, but the remaining ones must must be downloaded.
By default Pkg requests missing resources from package servers hosted by the JuliaLang organization at https://pkg.julialang.org: a global distribution network of over 50 servers spanning 10 server regions and 5 continents. Each server runs an instance of PkgServer.jl which is an open source reference implementation of a package server written in Julia.
In this talk we will take a detailed look at package server infrastructure. We will discuss how Pkg interacts with the servers (the simple package server protocol) and the implementation of PkgServer.jl. We will learn how new package releases are routed from developers to users. We will take a look at how https://pkg.julialang.org is designed and configured in order to minimize data transmission and latency, and show some interesting data and statistics gathered from the server network. Finally we will discuss how one can setup a custom package server for a company, a compute cluster, or simply for your local machine.
Slides: https://docs.google.com/presentation/d/1UgRIHlrc3C2Li3YL0kwnnbXMv31nXXLY1bR-Y2F0_sQ/edit?usp=sharing
Software Engineer at JuliaHub.