Marco Molinaro
Sessions
The ViaLactea Knowledge Base (VLKB) was set up as the main database for the Vialactea project, dealing with galaxy astrophysics of the Milky Way. The Vialactea project started at the end of 2013 and the VLKB, as a set of data resources and services customised to the benefit of the project, was ready and used by the end of 2015. The custom interfaces were defined keeping in mind the discovery and access scenario that is continuously developed in the Virtual Observatory (VO) ecosystem.
Interoperability was slowly brought inside the VLKB afterwards, depending on the limited resources available after the end of the Vialactea project. Nonetheless, the VLKB resources continued to be used in galactic astrophysics projects, and as a comprehensive resource of data and services in demonstrator projects. This helped the full system to be kept alive and updated (even if occasionally rather than continuously).
Currently, among the standards that are in use within the VLKB, an ObsCore table keeps the metadata for the observational datasets catalogue, a TAP service exposes the general underlying metadata content for all its data resources (catalogues, images, radial velocity cubes and morphologically complex objects, …), a custom implementation of the SODA standard is set up to enable dataset cutouts, and UWS is used to manage asynchronous cutout and merge requests. Furthermore, OAuth/OIDC AAI solutions have been tested on top of the cutout access service, and a multi-cutout solution has been presented at an IVOA meeting as a feedback to DataLink evolution. Other features, like management of complex morphology (tessellation, cross match, …), of simulated data, proper registration of the VLKB resources in a VO Registry, and more, are still missing or incomplete.
A dedicated client application to consume the VLKB, the ViaLactea Visual Analytics (VLVA), exists; the usage of standards is meant both to let the client be more general and easier to maintain and to enable generic client applications to connect to the VLKB resources.
For a smoother user experience and maintenance of the full VLKB system (e.g. to the benefit of the ECOGAL project and the contribution to the SKA-RC activities), the resources and services will however need to continue to mature and improve.
This contribution reports the status of the actual technologies and standards in use in the VLKB, and the future perspectives for the VLKB resources and services.
Prototyping access from visualisation tools to SKA science images and cubes stored in a rucio DataLake through IVOA discovery and access services.
M.Allen, R.Barnsley, M.Baumann, F.Bonnarel, T.Boch, C.Bot, R.Butora, J.Collinson, P.Fernique, V.Galluzzi., R Joshi, M.Molinaro, M. Parra-Royon, J. Sanchez-Castaneda , S. Sanchez-Exposito, G.Tudisco, F .Vitello A.Zanichelli.
SKA is the major low frequency radioastronomy project of the future with several major scientific applications: It will upgrade the amount of available science data by several orders of magnitudes reaching eventually more than 700 petabytes of storage per year. The SKA observatory will proceed to the initial data processing to deliver observatory data products while the SKA Regional Center network (SRC) will provide storage for those and processing capabilities to deliver and store advanced data products for the user community.
Within the scope of the SRC network, Orange (visualisation), Magenta (data management) and Coral (node implementation) teams have prototyped the discovery acces and visualisation of science data. Our visualisation tools VisiVO and Aladin discover, access and visualize test science data produced by SKA pathfinders stored in the rucio DataLake. Science metadata functionality has been implemented by the Magenta team to the Rucio data lake prototype to demonstrate a means of enabling IVOA-compliant data discovery and server-side processing.
VisiVo, Aladin Desktop and Aladin Lite are able to query the Discovery service built on ObsCore and SCS IVOA protocols.
This allows them to load DataLink responses providing links towards a SODA cutout service developed by the Orange team able to extract subcubes or images directly from the datasets stored in the rucio DataLake.
The Rucio Storage Element and SODA developments have been deployed and configured on the Spanish SRC node, providing computing and storage resources, managed by the Coral Team members. This prototype paves the way to collaborative development in the SKA regional center network and shows the possible integration of VO services and visualisation tools in DataLakes and science platforms.
The Trieste Solar Radio System (TSRS) was a set of two multi-channel solar radio polarimeters, performing continuous surveillance of the decimetric and metric coronal radio emissions with high time resolution. It was operational in Trieste (Italy) from 1969 to 2010, collecting data in digital form since 1999. However this archive encountered challenges in terms of maintenance and support due to resource constraints, including funding and personnel shortages.
More recently, the heliospheric physics community highlighted the importance of exploiting this resource, pushing the evaluation of new data preservation strategies with the primary goal to enhance accessibility and make the TSRS Heritage Archive (TSRS-HA) more adherent to the FAIR principles.
During the TSRS-HA setup process, the entire repository of original digitised raw data will undergo a complete re-ingestion, moving from bespoke data formats and proprietary software to standardised and open-source solutions. Consequently, this upgrade will lead to significant enhancements in data exploitation capabilities, addressing the limitations that were inherent in the former system and allowing users to conduct fast searches across the entire time series stored within the relational database.
The new architecture of TSRS-HA is based on a containerized microservice solution that breaks down the large application into smaller components, each enclosed within its own container. This introduces isolation allowing to change areas of the application without affecting the whole setup, improving security, providing fault isolation. Moreover, it supports portability and scalability, facilitating migration to new servers or cloud-based solutions.
The designed architecture allows accommodating multiple FAIR-enabling standards from different communities, like TAP, EPN-TAP and HAPI. The choice of multiple standard interfaces is driven both by their ability to connect to a larger audience and by the features they enable. The IVOA Table Access Protocol is chosen for its flexibility in deploying tabular data and rich metadata, as well as its support for metadata models. One of these models is the EPN-TAP to serve predefined atomic datasets alongside full historical time series. At the same time the Heliophysics Application Programming Interface can offer a solution dedicated to time series discovery and access.
Another benefit of employing containerized microservices is the availability of applications, like Jupyter notebooks, without the hurdles of conflicting frameworks on the host server. Jupyter notebooks can be used to provide practical examples and documentation for the services, offering users a cookbook-like introduction to leveraging the resources of TSRS-HA
This contribution provides technical details of the system and discusses future perspectives and potential refinements.
The Italian National Institute for Astrophysics (INAF) manages three single dish radio telescopes (Medicina, Noto and Sardinia Radio Telescope, SRT). The three dishes are also part of the European VLBI Network and the International VLBI Service for Geodesy & Astrometry. Also, SRT is involved in international collaborations dedicated to pulsar observation, namely the European Pulsar Timing Array and the Large European Array for Pulsars project.
The increasing importance of Science Archives and archive mining in defining the ultimate productivity of an observing facility motivated the Italian Centre for Astronomical Archives (IA2) service to develop and maintain the INAF radio data archive. Such a geographically-distributed archival facility flexibly handles different data models and formats, also supporting data discovery/access through Virtual Observatory (VO). In this contribution I will give an overview of the archival system, focusing on dealing with the increasing data rates/volumes produced by time-domain observations with state-of-the-art digital backends. I will address issues posed by the standardisation of time-domain-related data formats under the perspective of metadata completeness, necessary for archival publication. Also, I will present the INAF effort in modeling such data to enable their discoverability through VO tools and services.
Besides publishing radio data from the Italian radio telescopes, IA2 is also committed to provide access to data from international facilities and projects (such as ALMA data from ESO CalMS and Additional Representative Images for Legacy, ARI-L). I will finally mention the IA2 roadmap towards a modern Science Gateway, allowing users to produce advanced data products starting from telescope raw observations.