Astronomical Data Analysis Software & Systems XXXIV

Hugo Buddelmeijer

Bringing the same amount of rigor to the data processing as we have in the data analysis.


Session

11-12
14:45
15min
A data model to connect the ESO Data Processing System (EDPS) to ELT data archives
Hugo Buddelmeijer

C͢o͢n͢t͢e͢x͢t͢
The METIS and MICADO instruments for ESO's Extremely Large Telescope passed their Final Design Review mid 2024. That heralds an era of new data flow challenges for optical to mid-infrared astronomy. Reasons: (1) the instrumental complexity of the Adaptive Optics assisted ELT and its PSF Reconstruction, (2) nightly data rates up to several Terabytes, already during the laboratory testing phase. To address the laboratory testing phase challenge for METIS we will build an METIS AIT (Assembly Integration Testing) Archive database. The database contains the metadata of data items plus a pointer to the bulk FITS files. An interface between this database and the ESO Data Processing System would allow it to initiate processing laboratory data by querying the database. A similar approach could be adopted for MICADO.

T͢a͢l͢k͢
In this talk we present a prototype AIT Archive populated with simulated data. The archive is based on a data model. This model contains formalised descriptions of data items, of data processing recipes and pipelines (see [Buddelmeijer et al 2020]). The database tables are generated automatically from this data model.

We specify an interface between the ESO Data Processing System (EDPS) and the database. The interface is currently one-way: processed data can be automatically ingested into the database. We outline the plan to make the interface two-way such that the EDPS can also automatically retrieve necessary data.

Similarly, we specify an interface between the EDPS and the simulator. This ensures 𝗮𝗻 𝗲𝗻𝗱-𝘂𝘀𝗲𝗿 𝗼𝗻 𝘁𝗵𝗲 𝗘𝗗𝗣𝗦 𝗰𝗮𝗻 𝗿𝗲𝘀𝘁𝗿𝗶𝗰𝘁 𝗶𝘁𝘀𝗲𝗹𝗳 𝘁𝗼 𝘀𝗽𝗲𝗰𝗶𝗳𝘆𝗶𝗻𝗴 𝘁𝗵𝗲 𝗱𝗲𝘀𝗶𝗿𝗲𝗱 𝗲𝗻𝗱 𝗱𝗮𝘁𝗮 𝗽𝗿𝗼𝗱𝘂𝗰𝘁 𝗮𝗻𝗱 𝗹𝗲𝗮𝘃𝗲 𝗶𝘁 𝘁𝗼 𝘁𝗵𝗲 𝘀𝘆𝘀𝘁𝗲𝗺 𝘁𝗼 𝗱𝗲𝗰𝗶𝗱𝗲 𝗶𝘁𝘀𝗲𝗹𝗳 𝗵𝗼𝘄 𝘁𝗼 𝗮𝗰𝗾𝘂𝗶𝗿𝗲 𝘁𝗵𝗲 𝗻𝗲𝗰𝗲𝘀𝘀𝗮𝗿𝘆 𝗶𝗻𝗽𝘂𝘁 𝗱𝗮𝘁𝗮: 𝗿𝗲𝗮𝗱 𝗳𝗿𝗼𝗺 𝗹𝗼𝗰𝗮𝗹 𝗱𝗶𝘀𝗸, 𝗿𝗲𝘁𝗿𝗶𝗲𝘃𝗲𝗱 𝗳𝗿𝗼𝗺 𝘁𝗵𝗲 𝗱𝗮𝘁𝗮𝗯𝗮𝘀𝗲, 𝗼𝗿 𝗴𝗲𝗻𝗲𝗿𝗮𝘁𝗲𝗱 𝘁𝗵𝗿𝗼𝘂𝗴𝗵 𝘁𝗵𝗲 𝘀𝗶𝗺𝘂𝗹𝗮𝘁𝗼𝗿.

We outline the plan for these two interfaces, present the progress towards achieving it, and discuss how they will allow this AIT setup to evolve into a platform to support ELT data flow during ELT science operations (i.e., a “ELT Research Data Platform”).

Metadata, Semantics and Data Models Applicable to Data Formats
Aula Magna