2024-08-29 –, Room 6
In this talk we're introducing a new tool to build conda packages. It has been adopted by the conda community and is being rolled out in the widely used conda-forge distribution. The new recipe format has been vetted in multiple Conda Enhancement Proposals (CEPs). We are going to introduce the exciting new features of rattler-build (reproducible builds, high speed build execution, etc.). Using some examples, we will then discuss how you can use rattler-build & conda-forge to build highly optimized packages with SIMD and CUDA support. We will also take a look at cibuildwheel and recent improvements in the PyPI space for CUDA.
This talk has two aims: introducing a new tool – rattler-build – and showing how rattler-build is used to build highly optimized packages for the widely used conda-forge distribution (which ships thousands of scientific Python packages).
rattler-build has been developed from scratch to replace conda-build. It's a package build tool written in Rust, on top of the rattler libraries. We have undergone a meticulous standardization process through a number of Conda Enhancement Proposals. The result is a refined recipe format that removes a lot of the warts of conda-build:
- proper YAML - no semantic comments or arbitrary Jinja allowed
- integrates nicely with VSCode and other editors because we have a proper JSON schema
- strictly defined behavior for multi-output recipes
The result is that rattler-build is infinitely faster than conda-build at evaluating recipes. rattler-build also comes with perfect reproducibility for reproducible, bit-by-bit equivalent packages.
In conda-forge, there has been recent innovation to ship more optimized packages. In the past, packages that wanted to make use of SIMD features in the CPU had to implement "dynamic dispatching" at runtime - at the cost of a larger package size.
Since some time, conda-forge defines multiple "cpu-levels". These are defined for sse, avx2, avx512 or ARM Neon. On the client-side the maximum CPU level is detected and the best available package is then installed. This opens the doors for highly optimized packages on conda-forge that support the latest CPU features.
We will show how to use this in practice with rattler-build.
For GPUs, conda-forge has supported different CUDA levels for a long time, and we'll look at how that is used as well.
Lastly, we also take a look at PyPI. There are ongoing discussions on how to improve support for wheels with CUDA support. We are going to discuss how the (pre-)PEP works and synergy possibilities of rattler-build and cibuildwheel.
Discover the new rattler-build, and how to build optimized packages for modern CPUs and GPUs on conda-forge.
Category [High Performance Computing] –Other
Expected audience expertise: Domain –some
Expected audience expertise: Python –none
Public link to supporting material – Project Homepage / Git –
Wolf has been in open source for his entire professional career. He is a core member of conda-forge for the past 4 years. He started the mamba package manager (a C++ reimplementation of conda). He is also a steering council member for the conda project. He founded prefix.dev to take software package management to the next level. At prefix, pixi and rattler-build are developed to make shipping software a breeze.
