The DESC and the Rubin Software and Data Ecosystem
The start of operations at the Vera C. Rubin Observatory will herald a new era in astrophysics and cosmology, and with it, a new set of challenges. How can science collaborations deal with searching and sorting the large volume of data from Rubin, particularly as the project’s pipeline (and therefore the data’s provenance) evolves continuously over the course of the 10-year survey? How do we handle faulty data, or quantities for which multiple estimates exist from different sources? How do we provide meaningful value-added data products to the community, if these also require dedicated HPC resources to use to replicate analyses? How do we meaningfully interpret what our pipelines are even doing when an increasing number of their components are complex AI algorithms that may be soon be developed by other complex AI algorithms. DESC is beginning to grapple with these questions, and I will present some of the approaches we are taking in various DESC projects, and solicit your input for what we can do better.
