Karl Kosack
Data processing And Preservation Coordinator and Data Model Working Group chair for CTAO (Cherenkov Telescope Array Observatory). I'm also a staff astrophysicist at CEA Paris-Saclay Astrophysics Department.
I'm an astrophysicist that has worked for over 20 years on the detection of very-high-energy gamma rays, from hardware to software, and on the physics of objects that emit such radiation in the universe. I'm particularly interested in galactic particle accelerators and the origin of cosmic rays.
Session
We present the design and status of the data processing and data preservation software and infrastructure for CTAO, the next-generation very-high-energy (VHE) gamma-ray observatory under construction in Chile and the Canary Islands. The unprecedented size and complexity of CTAO and the need for open and reproducible data present new challenges for the storage and treatment of data products. We have developed open pipeline software to process petabytes of data generated by the observatory after on-site data volume reduction, including the treatment of tens of petabytes of simulated data required to model the instrumental response. This software produces reduced, standard data products usable by observers, in a format aimed to be compatible with other high-energy observatories. The data will be processed in parallel, data-driven workflows across six CTAO data centers (two on-site and four off-site in Europe). We will describe the big-data infrastructure we use for ensuring the preservation of the raw data and for executing the processing workflows. The technologies include Rucio, FTS, CVMFS, and significant contributions to expand the capabilities of the DIRAC middleware for complex workflows. We will also discuss our efforts to ensure both open software, standardized data models and formats, and FAIR data products. This includes also the ongoing efforts to create standard data models and formats for high-level data products that allow interoperability of CTAO data with existing X-ray, gamma-ray and neutrino observatories (i.e. GADF and VODF), as well as extending IVOA standards to meet the needs of this waveband for discoverability.