Juliacon 2024

Algebraic multigrid (AMG) methods [1] are among the most popular techniques to solve large systems of linear algebraic equations on supercomputers. They are widely used in the computational solid and fluid dynamics communities to solve problems with several millions of unknowns in parallel. Even though AMG schemes are key in many high-performance computing (HPC) applications, there are not many Julia alternatives available. AMG methods have been implemented in pure Julia in AlgebraicMultigrid.jl [2], which is a very valuable tool for the Julia community, but its implementation is sequential and thus not suitable for large-scale problems on supercomputers. Parallel AMG schemes remain available almost exclusively in packages with a long history such as PETSc [3] and Hypre [4], and implemented in traditional HPC languages like Fortran and C/C++. Even though these packages can be used from Julia via bindings (see e.g. HYPRE.jl [5]), a modern Julia implementation of parallel AMG methods is still missing. Without this implementation, the Julia package ecosystem misses a key feature in its otherwise rich scientific computing toolkit.

In this talk, we present the latest additions to PartitionedArrays.jl [6]. The package was initially introduced in JuliaCon 2022, and we want to share the key updates introduced since then with our users and the broader Julia community. In particular, we will focus on the implementation and usage of parallel AMG methods and other large-scale linear solvers in Julia.

The goal of this work is not to merely translate existing implementations of AMG into Julia, but to addresses some of their key limitations. Virtually all large-scale linear solvers are based on the Message Passing Interface (MPI) for parallelization. While MPI provides an efficient communication layer, it makes challenging the development cycle, specially in Julia. When debugging MPI code, a portion of the bugs can be tracked with conventional tools in a single MPI rank, but genuine parallel errors need to be debugged in parallel. Fixing an application that requires at least 100 ranks to crash can be very challenging, even with state-of-the-art (often proprietary) MPI debuggers. The situation is even worse in Julia. MPI codes are executed in batch mode with a fresh Julia session at each run. This makes difficult to reuse compiled code between runs, which is a major problem if your Julia code takes few minutes to compile from scratch. In addition, packages like Debugger are hard to use in combination with MPI on several processes.

For this reason, PartitionedArrays.jl introduces an alternative parallel programming interface that aims at solving these problems. This interface exposes a set of communication directives, which are defined as simple operations on arrays (hence the name PartitionedArrays.jl). These operations are implemented both for standard (sequential) Julia arrays as well as for MPI-distributed arrays. One can rely on standard arrays (and thus standard Julia tools) in the development and debugging process, and then change to MPI-distributed arrays when deploying large production runs in HPC clusters. This interface has been proven successful to implement distributed linear algebra data structures (vectors and sparse matrices), and key computational kernels (distributed sparse matrix-vector product). In this talk, we will show how it can also be exploited to implement AMG schemes and other important functions such as distributed sparse matrix-matrix multiplication kernels. With this, we aim at simplifying the development process of such complex algorithms.

References

[1] https://doi.org/10.1016/S0377-0427(00)00516-1
[2] https://petsc.org/release/
[3] https://computing.llnl.gov/projects/hypre-scalable-linear-solvers-multigrid-methods
[4] https://github.com/JuliaLinearAlgebra/AlgebraicMultigrid.jl
[5] https://github.com/fredrikekre/HYPRE.jl
[6] https://github.com/fverdugo/PartitionedArrays.jl