ADASS 2022

CANFAR: A Community-Built Astronomy Platform
2022-11-02 , ADASS Conference Room 1

The size and complexity of data from new observatories continues to put science platforms in the spotlight. The Canadian Advanced Network for Astronomical Research (CANFAR) began in 2008 and has been through many changes, but none more significant as the transition from virtual machines to containers in 2018, resulting in the CANFAR Science Platform. With this transition came a number of challenges, but also shifts in our way of thinking how an astronomy platform can be built.

CANFAR is a general-purpose astronomy platform, used by projects from a variety of facilities such as the James Webb Space Telescope (JWST), the Atacama Large Millimeter Array (ALMA), the Rubin C. Vera Observatory, and many others. One of the difficulties users face in the original virtual machine-based cloud is a steep technical learning curve to get a simple science function operating. The model is now quite different: users are offered a choice of a variety of ready-to-use containers. A small, core set are provided and maintained by us, but the user community has contributed the majority. Shifting this responsibility has allowed us to focus on platform-wide improvements that benefit all projects. The community brings value to the platform by providing shared astronomy software containers.

A science container is in one of several categories: Jupyter Notebooks, CARTA (Cube Analysis and Rendering Tool for Astronomy), X11 Desktops (ARCADE), and another, general web-interface category. These allow for the accounting of a small set of differing requirements. When launched, a science container becomes an active session which users interact with through their browser. When using an ARCADE Desktop session, a variety of other non-web based containers can be run, whose displays are attached and displayed on the desktop. A user-managed image registry hosts the set of containers available for running on the platform.

This self-serve model is a common theme in the platform. Project leads create groups representing their teams. Using those groups, they give teams access to not only the software images, but also to the shared, distributed storage that is available on containers. Many of these concepts have been driven by our participation with the International Virtual Observatory Alliance (IVOA).

Though we feel we have a sound and sustainable model and architecture, our efforts have not been without challenges. The adoption of kubernetes was difficult, but in the end well worth it. Also, some usability problems are still hard, such as efficient resource management and storage latency. We have a long list of features and improvements to make, which includes support for different types of batch processing.

Canadian Astronomy Data Centre (CADC)