Welcoming remarks delivered by POC and LOC chairs

This contribution presents our concept for collaborative and guided visual analytics of astrophysical and planetary data developed within the EXPLORE EU project (https://explore-platform.eu). We implement and evaluate new concepts using Visualizer [1], our web-based open-source visual analytics framework. Visualizer is designed for rapid prototyping of research applications, with easy-to-use extensibility and integrability mechanisms built into its core. Adding new visualizations is as simple as implementing a small JavaScript API and uploading the visualization code to the running system. This enables software developers but also researchers with little programming experience to easily integrate custom visualizations. Similarly, new data analysis algorithms can be integrated by wrapping them with an algorithm-agnostic execution REST-API. An API enables users to easily integrate a dashboard designed in Visualizer into any web-based application. In this work, Visualizer is used for development of astrophysical and planetary scientific data applications (SDAs) which are currently being extended to provide collaboration and user guidance methods.

Initial collaboration features enabled users to easily share visual analysis interfaces by exchanging URLs or QR codes. In addition, real-time collaboration methods make it possible for users to collaboratively explore their data or even design visual interfaces together. However, literature research, e.g., [2], as well as requirements from the EXPLORE SDAs revealed that data annotation methods represent another major requirement for collaborative visual analytics. Annotation features enable users to mark and describe any data of interest within visualizations, as well as to share findings and rate each other’s annotations. These features also provide a discussion platform enabling domain experts to collaboratively explore their data, discuss interesting findings, and exchange opinions. For non-experts annotations provide a source of additional knowledge enabling them to better understand complex data sets, relationships, or correlations. For machine learning algorithms the annotations might be used as (additional) training data to improve their performance.

To support the data exploration process, we propose user guidance methods for suggesting and (optionally) automating analytical workflows. Typically, to obtain the desired analytical result the user manually selects and configures various algorithmic and visual data analysis steps and applies them. Given an analytical goal (e.g., outlier detection or correlation analysis), our guidance system supports novice users by recommending analytical workflows that consist of several analytical steps. Relying on provided result previews, the user can select and execute a whole workflow. Alternatively, a more advanced user can reconfigure (parts of) a recommended workflow by manually changing and parametrizing algorithms and visualizations. To offer user guidance, an AI method learns how (experienced) users analyze data and predicts analytical steps for other users. Different approaches have been investigated within our research framework [3] including Markov chains [4] and deep learning approaches. Both approaches learn from user behavior, which is either collected implicitly, e.g., user interaction logging, or explicitly, e.g., user ratings of analytical results.

EXPLORE will deliver six SDAs involving galactic stellar and lunar data, with collaborative visual analytics contributions planned for most of them. For example, we integrated a new 3D volumetric visualization and used it in the G-Tomo SDA (Fig. 1) to represent extinction data cubes. The 3D cube visualization was extended with the interaction functionality supporting creation of 2D slices which, using the Visualizers view coordination framework in the background, are then shown in the contour plot. In the guided analytics example (Fig. 2) the user selects an analytical goal - in this case anomaly detection – and the framework suggests analytical workflows, i.e., sequences of algorithmic and visual methods, that other users employed to achieve that goal. In the S-Phot SDA (Fig. 3) we show the spectral energy distribution of stars using scatterplots and employ a stellar atmosphere model fitting algorithm provided by a consortium partner. Here, annotation is employed to mark some of the outliers (gray data points) as requiring further investigation. In addition, the results of the data analysis can be shared with other users and explored together in real time (Fig. 4).

Acknowledgement

This project has received funding from the European Union’s Horizon 2020 research and innovation programme under grant agreement No 101004214.

Know-Center is funded within the Austrian COMET Program, under the auspices of the Austrian Federal Ministry of Transport, Innovation and Technology, the Austrian Federal Ministry of Economy, Family and Youth and by the State of Styria. COMET is managed by the Austrian Research Promotion Agency FFG.

References

[1] Ilija Simic (2018) Visualizer - An Extensible Dashboard for Personalized Visual Data Exploration, Master’s Thesis at Graz University of Technology

[2] M. Elias and A. Bezerianos (2012) Annotating BI Visualization Dashboards: Needs & Challenges, Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (CHI '12)

[3] B. Mutlu, E. Veas and C. Trattner (2016) VizRec: Recommending Personalized Visualizations, ACM Transactions on Interactive Intelligent Systems (TiiS)

[4] Milot Gashi (2018) Goal oriented Visual Recommendations via User Behavior Analysis, Master’s Thesis at Graz University of Technology

CDS (Strasbourg astronomical Data Center) celebrates in 2022 the 50th anniversary of its creation. 2022 also marks the 30th anniversary of the start of the Aladin project.
The original mission of the project - a sky atlas providing access to reference image surveys, enabling comparison of multi-wavelength data, cross-identification of sources - still holds strong. It it now fully supported by the HiPS (Hierarchical Progressive Survey) format.

We want to take the opportunity of this 30th anniversary to provide with a comprehensive summary of the HiPS ecosystem that CDS and its close partners (ESO, ESA, STScI) have built in the last decade, and which allows the creation, publication, visualisation and scientific exploitation of HiPS datasets.

As of July 1st 2022, the HiPS network is serving more than 1,000 HiPS with a total volume of 400TB and 300,000 billion pixels. Available HiPS are quite diverse, going from all-sky surveys (AllWISE, Integral, DSS, ...) to pointed observations (XMM-Newton, HST, HSC, Spitzer, ...), covering the whole electromagnetic spectrum and with a large variety of angular resolution.
HiPS is a container not only for pixel data, it is used to serve efficiently gravitational waves probability maps (LIGO/VIRGO), fluxes maps, density maps for VizieR catalogues, data cubes, but also large catalogues as the recent Gaia DR3 main table for instance.
HiPS have also been created for planetary surface data.

The large majority of the available HiPS images have been generated with the field-proven Hipsgen tool which was used notably to create HiPS for individual PanSTARRS bands (three quarter of the sky at 250mas/pixel resolution) in 16 days on a single server, for a total of 40 TB of FITS tiles.
The Montage toolkit, developed by IPAC, is also capable of generating HiPS individual tiles.

A network of 24 HiPS nodes is in use for distribution of the HiPS data, ensuring redundancy and good access time around the world.

HiPS data can be consumed by a dozen of existing clients: heavy clients (KStar, Stellarium, Aladin Desktop), web clients (Aladin Lite, World Wide Telescope, HscMap, Firefly) in various contexts: scientific purpose (with access to pixel real values), visualisation, amateur astronomy but also for EPO with the recent HiPS support in the Digistar planetarium control software.
Aladin Lite, developed at CDS, has been implemented in more than 50 professional astronomy websites and is being endorsed by ESA and ESO portals to explore and access their archival data.Version 3, currently in beta test, will offer all-sky projections, coordinates grid display, access to real pixel value (FITS tiles) and improved display thanks to GPU-based rendering.

HiPS is a FAIR (Findable, Accessible, Interoperable, Reusable) product: the HiPS protocol has been endorsed by the IVOA (International Virtual Observatory Alliance) and relies on proven existing standards (FITS, WCS, HEALPix). Individual HiPS are findable as resources in VO registries. As the data are sampled in a common space grid, they can be easily composed and reused, as shows for instance the hips2fits service which provides arbitrary FITS cutouts for any published HiPS.

The HiPS format itself has proven to be quite efficient and resilient to high load: in the hours following the release of the first JWST images, CDS servers have recorded more than 30 million requests on average (coming from 100,000 different IPs) with peaks at 1,500 requests per second.

We will conclude our presentation with an overview of the perspectives: access to HiPS and services related to large data cubes (in the context of SKA) and SED (Spectrum Energy Distribution) generation from photometric-calibrated HiPS.

Several years ago, IRSA embarked on the task of regularizing all our image and spectral metadata by converting them to the CAOM data model. Our goals are to reduce the number of custom tools needed to support data ingestion and user interfaces, standardize our internal and external interfaces on VO services and protocols, and enable ObsCore/ObsTAP data access. We will describe the progress we've made, the hurdles we're still struggling with, and lessons learned about how to approach changing the data model layer underneath a petabyte-scale (and growing) data archive spanning dozens of diverse datasets.

Poster block 1

The Science Archives implemented by ESAC Science Data Centre (ESDC) are the final data repositories of the ESA missions. Their software architecture has experienced significant changes since the first archive was released in 1998. The evolution resulted in several generations of archives adopting the latest state-of-the-art technologies and improving the user experience dramatically.

The Integral Science Legacy Archive (ISLA) is one of the newest archives in which special effort has been dedicated to engaging the mission scientists to develop a science-oriented archive. In addition, special care has been taken in offering the best user experience moving from the classical data-forms flavour to dedicated Integral Science Portals. In this talk, we will walk through those Integral specific sections and see how this approach has inspired other ESDC archives.

The implementation of ISLA has been driven by the UX design paradigm and the adoption of mature software tools like Angular. This Javascript framework promotes highly performant and usable applications. It can be considered a dedicated platform that supports the implementation of responsive, efficient, and high-quality cross-platform solutions.

In the talk, we will see the importance of being up-to-date and using the latest technologies in order to get powerful and maintainable tools that fulfil complex science use-cases.

ADASS is often referred to as “A gathering of old friends, rather than a
scientific conference”. This statement is generally used as a way to articulate
the sentiment that the members of the ADASS community are, on average,
friendly and easily accessible and why ADASS is generally a very nice meeting
to attend. However, being a gathering of old friends can make it hard for
newcomers, without existing connections in the community, to be fully included.
To make the community more accessible for junior staff members and other
newcomers, we propose to organise a free-flowing BoF session, following the
simple rule that newcomers should be stimulated to discuss any relevant topic
with each other and/or the established members of our community. We en-
vision those topics to range from, for instance, ways to navigating the work
floor, how to pitch more modern programming languages, or how to engage in
collaborations.
To give the novice attenders an idea of what ADASS is, we propose 3 vet-
erans giving a pitch for about 5min about the history and their own personal
experience to kick-start discussions and/or fill an online board with questions
they can answer.

Astronomical data are getting bigger and bigger. They’re so big now that it’s completely impractical for researchers to obtain local copies of many important datasets, pushing us into the the age of “science platforms”: instead of running analysis software on local hardware, researchers do their work by interfacing with powerful remote systems. This paradigm shift creates both new opportunities and new challenges. A major challenge has to do with data visualization: existing astronomy data visualization tools are often desktop applications designed for local datasets that are small, in the sense that they must fit in a single computer’s memory. In the age of the science platform, visualization applications need to handle huge datasets, and they need to provide a seamless user experience when such datasets live on remote servers. Fortunately, we have a great software platform for building such applications: the web browser. Modern browsers are tools that power complex, interactive, network-native, multimedia applications, thanks to billions of dollars of industry investment. Indeed, the rise of the science platform is closely tied to the development of JupyterLab, a web-native framework for interactive computing. This tutorial will introduce participants to the tools and concepts that underpin modern, browser-based interactive astronomical visualization software, including aspects relating to the visualization user experience (UX) design. It will do so while working through the steps to visualize very large (gigapixel and beyond) FITS imagery interactively in JupyterLab using new features introduced in the “2022 edition” of AAS WorldWide Telescope (WWT) and its related tooling.

How do you decide whether it’s better to adopt or adapt an existing package or build something completely new? What are the implications of outsourcing vs maintaining one’s own code? In this talk I look at how these decisions, perennial concerns in software engineering, are made in the astro community. Using several real world examples, I examine some common ways of assessing the tradeoffs with a view to minimizing long-term technical debt and getting out of the way of doing excellent science.

Many of the tools we develop build depend on core numerical libraries such as FFTW, or more generally ecosystems such as those from netlib or scipy. Whilst in many cases these core libraries are well regarded and used, the need to either run tools in new environments (such as client-side in a web browser or on a mobile device), or with the rise of new challengers to the current Fortran/C/C++ ecosystem, such as Rust, Julia or possibly even Go, means that either these libraries are being ported, or new libraries being written wholesale. This BoF aims to start the conversation around what we can do to ensure these new libraries are trustworthy, by firstly covering some of the experience of the BoF organisers, and then opening up a wider discussion.

Software testing, in all its forms, is a crucial part of the software development process. It can take a significant fraction of time and effort but its perceived importance (and therefore allocated funding) is often minimal in all but very large projects. Automated CI/CD systems, for example, are a useful set of tools to streamline and automate some of this important task, increasing testing coverage and confidence in our product, and in turn freeing developers to do more creative work. Nevertheless, as developers we often still find ourselves doing more donkey work in this space than we would prefer, or - worse - decreasing the amount of testing to meet a deadline or create the illusion of decreasing project cost.
This BoF will discuss how we test our software: what works, what doesn’t for small and large projects, and how we can use our experiences to help each other to both improve testing and minimise the effort required to test well. We will encourage participants to pose questions, offer suggestions, and even air dirty laundry in a judgement-free space.

The Astrophysics Source Code Library contains over 2700 metadata records on astro research
software. This short hands-on tutorial is geared to new users of the resource, though even
advanced users of it are sure to learn something new.
Attendees will learn:
* alternate ways to bring up ASCL records
* how to find software
* Hands-on activity: Search for software using at least three different methods:
* full-text search
* ASCL’s API
* ADS and Google
* how citation tracking and preferred citation work
* how to create a metadata file for their own software that tells others how to cite their code
* Hands-on activity: Create codemeta.json and CITATION.cff files using ASCL and GitHub
* Hands-on activity: Add preferred citation information to their own software
* what metadata is contained in the ASCL and how it is structured
* how to download the ASCL’s contents for their own projects
* Hands-on activity: Experiment with ASCL metadata using some simple tools
* how to submit software to the ASCL
* Optional hands-on activity: Submit a code to the ASCL

The tutorial will cover related topics, including the differences between ASCL and ZENODO, and
how, why, and when to submit to each, as time permits.

The Gamma-ray Coordinates Network (GCN) is a public collaboration platform run by NASA for the astronomy research community to share alerts and rapid communications about high-energy, multimessenger, and transient phenomena. Over the past 30 years, GCN has helped enable many seminal advances by disseminating observations, quantitative near-term predictions, requests for follow-up observations, and observing plans. GCN distributes alerts between space- and ground-based observatories, physics experiments, and thousands of astronomers around the world. With new transient instruments from across the electromagnetic spectrum and multimessenger facilities, this coordination effort is more important and complex than ever. We introduce the General Coordinates Network, the modern evolution of GCN built on modern, open-source, reliable, and secure alert distribution technologies, and deployed in the cloud. The new GCN is based on Apache Kafka, the same alert streaming technology that has been selected by the Vera C. Rubin observatory. In this talk, we will present the status and design of the new GCN, a tutorial on how to stream alerts, and a vision of its growth as a community resource in the future.

SIPGI (2022MNRAS.514.2902G) is a software to reduce optical/near-infrared spectroscopic data acquired by slit-based spectrographs. The software provides a GUI written in Python, used to organise data, to run the reduction recipes (written in C) and to check reduction results with a set of interactive graphical tools. The interaction between Python and C is obtained using the SWIG wrapper.

SIPGI is a complete spectroscopic data reduction environment based on three key points:
1. the instrument model, i.e. the analytic description of the main calibration relations: spectra location, spectra tracing and wavelength calibration. This model can be easily checked and adjusted on data by using a graphical tool;
2. a built-in data organiser that classifies the data and, together with a graphical interface, helps in feeding the recipes with the correct inputs;
3. the design and flexibility of the reduction recipes; the number of tasks required to perform a complete reduction is minimised, while preserving the possibility of verifying the accuracy of the main stages of the data-reduction process.

This architecture allows SIPGI to retain the high level of flexibility and accuracy typical of the standard 'by-hand' reduction and, on the other side, to provide a higher level of efficiency.

The current version of SIPGI manages data from the MODS and LUCI spectrographs mounted at the Large Binocular Telescope. It is our plan to extend SIPGI to other through-slit spectrographs.

In the past dozen years, the ecosystem around astronomy software has seen great changes, including improved infrastructure for creating, managing, indexing, storing, and archiving code, more journals in which to publish software, and better citation methods and citation tracking of computational methods. What changes are coming up? How can we build on the progress that’s been made? What further improvements are needed? This BoF will have an open discussion with attendees, with a panel to guide the conversation, to explore what additional changes are needed and what paths can bring these ideas to fruition.

For software which is useful to people other than its producer by itself or as a generator of data, it may be useful to publish an article about how the software works in a refereed journal. This is a great way to publish and publicize our software efforts, and get publication and citation credit for our work, supplementing software distribution sites and online documentation. A panel of representatives of various journals which publish articles about software will discuss what they are looking for in papers and how to participate in the publishing process, not just as an author, but as a referee.

In this presentation we will cover some astronomy design patterns and perhaps some anti-patterns in astronomy. We will use our experience on several long projects such as Rubin Observatory, Gaia, SDSS, UKIRT and JCMT to highlight some of the the things which worked and a few things that did not work so well. For example separation of data access from data format is common across all of our projects and something we find critical. The Rubin Science Platform (and deployment system) underpinned by infrastructure as code is also a culmination of many previous efforts.

Astronomy is experiencing an explosion in the number of observations of celestial objects due to large-scale imaging surveys aimed to understand the formation & evolution of many classes of objects. Our team of data scientists and astronomers is involved in two such surveys: KiDS and Euclid. Our objective is to extract samples of various rare types of objects with high completeness/recall and purity/precision from an ocean of up to billions of source detections. This often requires a laborious effort combining automated feature extraction followed by human inspection. We have encountered the limit of this approach for KiDS, while Euclid will increase the challenge by an order of magnitude. In this paper we report on experiments to simultaneously identify 3 classes of rare objects (galaxy mergers, strong gravitational lenses and asteroid streaks) via a tiered cascade of Convolutional Neural Networks.

Qserv is a distributed, shared-nothing, SQL database system being developed by the Vera Rubin Observatory to host the multi-petabyte astronomical catalogs that will be produced by the LSST survey. Here we sketch the basic design and operating principles of Qserv, and provide some updates on recent developments and scale testing.

We present the Rubin Observatory system for data storage/retrieval and pipelined code execution. The layer for data storage and retrieval is named the Butler. It consists of a relational database, known as the registry, to keep track of metadata and relations, and a system to manage where the data is located, named the datastore. Together these systems create an abstraction layer that science algorithms can be written against. This abstraction layer manages the complexities of the large data volumes expected and allows algorithms to be written independently, yet be tied together automatically into a coherent processing pipeline. This system consists of tools which execute these pipelines by transforming them into execution graphs which contain concrete data stored in the Butler. The pipeline infrastructure is designed to be scalable in nature, allowing execution on environments ranging from a laptop all the way up to multi-facility data centers. This presentation will focus on the data management aspects as well as an overview on the creation of pipelines and the corresponding execution graphs.

Poster block 2

A new generation of survey telescopes with large etendues (the product of field of view and
collecting area), are detecting and resporting variable astrophysical events in the form of large
astronomical alert streams. In particular, the Zwicky Transient Facility has been producing
approximately 300 k alerts per night since 2018 and the Vera C. Rubin Observatory will produce
about 10 M alerts per night since 2024. In order to make sense of these large alert streams a
new type of system is needed: the astronomical alert brokers. Several brokers have been
processing the alert stream from ZTF, and seven brokers have been selected as Community
Brokers for the Vera C. Rubin Observatory and its Legacy Survey of Space and Time (LSST):
ALeRCE, ANTARES, AMPEL, Babamul, Fink, Lasair and Pitt-Google. These brokers will
become intermediaries between survey telescopes and follow-up resources and will offer
different services for the general community to provide access to the alert stream and enable
the best science from these data. In this tutorial we will present the tools and services provided
by the ALeRCE broker.

The carbon footprint of astronomical research is an increasingly topical issue with first estimates of research institute and national community footprints having recently been published. As these assessments generally do not take into account the contribution of astronomical research infrastructures, we propose to complement them by providing an estimate of the contribution of astronomical space missions and ground-based observatories using greenhouse gas emission factors that relate cost and payload mass to carbon footprint. We find that use of astronomical research infrastructures dominates the carbon footprint of an average astronomer. Comparison of our findings with the socio-economic pathways that, according to the Intergovernmental Panel on Climate Change (IPCC), are compliant with keeping the global average temperature rise below levels of 1.5°C or 2°C suggests that drastic changes are needed on how astronomical research is conducted in the future. Specifically, continuous deployment of ever more and larger astronomical research infrastructures is clearly not sustainable. We argue that a new narrative for doing astronomical research is needed if we want to keep our planet habitable.

The ESO Data Processing System (EDPS) is a new software infrastructure to run ESO science data processing modules (“pipeline recipes”) that is currently in an advanced status of development. These recipes are used for quality control purposes, unsupervised production of science and calibration products for the archive, and are embedded in an interactive data reduction system that is also offered to external science users. EDPS aims to unify all these use cases and thereby replace all dedicated systems that are currently used at the European Southern Observatory (ESO).
EDPS is designed to handle all present and future instruments from ESO's La Silla, Paranal and, in perspective, Armazones (ELT) Observatory. They range from simple imagers, to 1D and 3D spectrographs to optical interferometry (VLTI), across the entire optical and infrared wavelength interval. Instrument-specific workflows can be configured in EDPS using a flexible DSL (Domain Specific Language) written in Python.
In this talk I will present the high-level requirements of EDPS as well as its design, focusing in particular on the data organisation and processing model and on the workflow specification language.

ESASky (sky.esa.int) is a tool to discover multi-mission, multi-wavelength and multi-messenger astronomical data from a wide variety of ground and space missions. In order to explore new ways to access the data contents of ESASky, as well as the contents from other astronomical data centres accessible from it, the ESA Virtual Assistant (EVA) has been integrated and released to the public in mid-2022. The ESA virtual assistant uses artificial intelligence and natural language processing to allow users to interact with ESASky’s API using simple sentences. At the moment the virtual assistant has been programmed with a sufficient number of interactions to carry out most of the tasks that a typical user would do in the ESASky web interface. However, exposing the virtual assistant to a larger audience will allow us to train the neural network running behind the virtual assistant to understand more complex sentences and commands, opening new avenues for more advanced exploration of ESASky.

Searching for binary pulsars is a computationally costly process that next generation radio telescopes will have to perform in real time, as data volumes are too large to store. In this talk we will explore how we have reduced the energy cost of an SKA-like Fourier Domain Acceleration Search (FDAS) implementation in AstroAccelerate, utilising a combination of mixed precision and dynamic frequency scaling on NVIDIA GPUs.

The goal of this talk will be to encourage you to consider exactly how much precision you need, as we have managed to save 50% of the execution time by using mixed precision with a completely tolerable (<3%) sacrifice in numerical sensitivity. With dynamic frequency scaling, the overall energy cost of FDAS was reduced by over 70%.

SpectraPy is an Astropy affiliated package for spectroscopic data reduction. It collects algorithms and methods for data reduction of astronomical spectra obtained by through-slits spectrographs. SpectraPy is the first brick of a Python environment for spectroscopic data reduction. It has been created to fill the gap in Astropy between the already existing file data handling libraries and those for spectra analysis.

SpectraPy combines Astropy facilities with SAOImageDS9 features, providing a set of tools for spectra calibration and 2D extraction. It starts from raw frames, and using configuration files which describe the optical setup of the instrument, it automatically locates and extracts 2D spectra that have been wavelength calibrated and corrected by distortions.

The library, designed to be spectrograph-independent, can be used on both longslit (LS) and multi object spectrograph (MOS) data. It comes with a set of ready-to-use configuration files for the LBT-LUCI and LBT-MODS spectrographs, but it can be configured for data reduction of other through-slits spectrographs

In the future we want to extend SpectraPy adding other classes and methods (e.g. background subtraction or 1D spectra extraction) needed to obtain fully reduced spectra.
Moreover we would like to improve SpectraPy in order to reduce data obtained from fiber spectrographs.

The size and complexity of data from new observatories continues to put science platforms in the spotlight. The Canadian Advanced Network for Astronomical Research (CANFAR) began in 2008 and has been through many changes, but none more significant as the transition from virtual machines to containers in 2018, resulting in the CANFAR Science Platform. With this transition came a number of challenges, but also shifts in our way of thinking how an astronomy platform can be built.

CANFAR is a general-purpose astronomy platform, used by projects from a variety of facilities such as the James Webb Space Telescope (JWST), the Atacama Large Millimeter Array (ALMA), the Rubin C. Vera Observatory, and many others. One of the difficulties users face in the original virtual machine-based cloud is a steep technical learning curve to get a simple science function operating. The model is now quite different: users are offered a choice of a variety of ready-to-use containers. A small, core set are provided and maintained by us, but the user community has contributed the majority. Shifting this responsibility has allowed us to focus on platform-wide improvements that benefit all projects. The community brings value to the platform by providing shared astronomy software containers.

A science container is in one of several categories: Jupyter Notebooks, CARTA (Cube Analysis and Rendering Tool for Astronomy), X11 Desktops (ARCADE), and another, general web-interface category. These allow for the accounting of a small set of differing requirements. When launched, a science container becomes an active session which users interact with through their browser. When using an ARCADE Desktop session, a variety of other non-web based containers can be run, whose displays are attached and displayed on the desktop. A user-managed image registry hosts the set of containers available for running on the platform.

This self-serve model is a common theme in the platform. Project leads create groups representing their teams. Using those groups, they give teams access to not only the software images, but also to the shared, distributed storage that is available on containers. Many of these concepts have been driven by our participation with the International Virtual Observatory Alliance (IVOA).

Though we feel we have a sound and sustainable model and architecture, our efforts have not been without challenges. The adoption of kubernetes was difficult, but in the end well worth it. Also, some usability problems are still hard, such as efficient resource management and storage latency. We have a long list of features and improvements to make, which includes support for different types of batch processing.

As data volumes continue to grow there is a need to move processing not only closer to the source of the data but also closer in time to when the data are created. This style of processing data as it streams off the telescope leads to data pipelines that need to focus on throughput. Bifrost is an open-source, modular C++/CUDA/Python framework that is intended to make building reliable, high-performance data capture and analysis pipelines such as this easier. In this talk I will discuss Bifrost and its core concepts of ring memory spaces and blocks how they are used for CPU and GPU computing. I will also discuss use cases for Bifrost by providing examples of instruments that are currently using or investigating using the framework. Finally, future development plans and directions for the framework will be presented.

The SKA Observatory will generate 700 PB of science ready data products per year. These will be made available to the astronomical community through a worldwide network of SKA Regional Centres (SRCs), which will be organised by local communities that bring together radio astronomy institutes and compute centres.

Science analysis platforms provide scientists with an interface to software and workflows, data and processing hardware in a uniform way. Generally an interface through a browser (e.g. using Jupyter notebooks) and APIs for programmatic access. The future SKA science analysis platform provided by the SRCs should allow complex distributed analysis and data exploitation including science pipelines, machine learning and other advanced techniques working on one of the most significant existing science data lakes and using a complex federated computational environment. This system should not be only performant and efficient but, also, these complex features should be presented to the community in an easy way. In preparation for this, the SKA Regional Centre Steering Committee (SRCSC) has initiated several prototyping activities to investigate how currently existing tooling from other instruments and fields could be leveraged to provide the infrastructure within which the SRCs will operate. In this contribution we present the work of the prototyping team focused on development of the SKA science analysis platform, which operatess under the name “Team Tangerine”.

Every field, and even every instrument within a field, is different, and therefore the definition of what a science analysis platform is and does cannot be defined in a uniform way. Because of this there are also many resources which either describe existing platforms, or more theoretically discuss what they typically should support. The first task of the Tangerine team is therefore to draft a vision of what this term means in the SKA context, based on literature study and the requirements from of the SKA. In parallel we make a qualitative comparison of the main platforms known to us and assess how an SKA platform could look, based on currently existing platforms or components. In this contribution we aim to present the vision on the SKA science analysis platform.

In the Jupyter ecosystem, data visualization is usually done with “widgets” created as notebook cell outputs. While this mechanism works well in some circumstances, it is not well-suited to presenting “ds9-like” interfaces that are long-lived, interactive, and visually rich. Unlike the traditional Jupyter notebook system, the newer JupyterLab application provides a sophisticated extension infrastructure that raises new design possibilities. Here we present a novel user experience (UX) for interactive data visualization in JupyterLab that is based on an “app” that runs alongside the user's notebooks, rather than widgets that are bound inside them. We have implemented this UX for the AAS WorldWide Telescope (WWT) visualization tool and will demonstrate it in operation. JupyterLab's messaging APIs allow the app to smoothly exchange data with multiple computational kernels, allowing users to accomplish tasks that are not possible using the widget framework. While we have developed this UX for WWT, the overall design is portable to other applications and has the potential to unlock a variety of new user activities that aren't currently possible in “science platform” interfaces.

Despite decades of developments, combining multiple ground-based astronomy exposures into high signal-to-noise coadds while also improving their spatial resolution is still an outstanding challenge today. Here, we present an expectation-maximization framework for multiframe image denoising and deconvolution, which can be readily extended for performing (1) blind deconvolution, where we self-consistently solve for the point-spread function of each exposure, (2) super-resolution, and (3) improved background subtraction. Our TensorFlow implementation is computationally efficient, scalable, and flexible, as it benefits from advanced algorithmic solutions and leverages Graphical Processing Unit (GPU) acceleration. The testbed for our method is a set of 4K Hyper Suprime-Cam exposures, which are closest to the quality of imaging data from the upcoming Rubin Observatory. Our analysis of the data reveals that performing denoising and deconvolution using traditional gradient descent-based methods tends to result in physically uninterpretable coadds, as the optimizers converge to bad local minima due to their greedy nature. We developed an extension that not only consistently enforces desired constraints, such as non-negativity of pixel values in the coadd, but also proceeds in a way that yields no ``usual'' artifacts. The preliminary results are extremely promising: unprecedented details such as the shape of the spiral arms of galaxies are recovered, while we also manage to deconvolve stars perfectly into essentially single pixels. Statistical tests of the extracted source catalogs are ongoing.

We are in the era of Big Data: the massive amounts of data generated in Astrophysics are, as of today, in the petascale if not already in the exascale. In the near future, we will see the amount of data grow further. Big physics projects like the High-Luminosity Large Hadron Collider (HL-LHC) and the Square Kilometre Array (SKA) are each expected to produce, in the next decade, an exabyte of data yearly. These amounts of generated data are constantly setting new challenges for data processing, reduction and analysis and new models are needed for data management and computation.
In Astrophysics, in the environment of the International Virtual Observatory Alliance a big work has already been done for what concerns data management. An architectural model, a set of standards to describe, retrieve and exchange data and data access services have been developed to build the global Virtual Observatory gaining complete data FAIRness.
The new challenges posed by big data processing, need software and hardware solutions overcoming the traditional ones and new models for resource access. In this talk I describe a resource access model based on the work done in the IVOA.

CASA, the Common Astronomy Software Applications, is the primary data processing software for ALMA and the VLA, and is frequently used for other radio telescopes. CASA can handle data from single-dish, aperture-synthesis and Very Long Baseline Interferometry (VLBI) telescopes. Based on two recent reference papers, this talk will give a high-level overview of the basic structure and functionality of the CASA software, with special emphasis on recent development work that made CASA fully functional for processing VLBI data.



CASA is a user-friendly package for the manual processing of radio astronomical data and formally supports complex calibration and imaging pipelines for the ALMA and the VLA telescopes. This has also led CASA to be the foundation of one of the two calibration pipelines used by the Event Horizon Telescope, as well as two community-driven MeerKAT science pipelines and one for the GMRT.

The CASA infrastructure has been evolving and it is now offered in a pip-wheel based package, allowing the use of CASA in personal Python environments. Moreover, EC funding has facilitated the development of a Jupyter-CASA kernel. These developments enable users to work with CASA in the Jupyter notebook environment, and can be used for training, tutorials, as well as basic calibration pipelines.

We will also briefly discuss the future of CASA within the context of the proposed ngVLA and upgraded ALMA telescopes, summarize our prototyping efforts based on modern parallel infrastructures such as Dask, and mention ongoing efforts regarding advanced algorithms and modern visualization tools. CASA is being developed by a large international team of scientists and software engineers based at the NRAO, ESO, NAOJ, and JIVE.

A Computer vision models based on deep learning are used in a wide variety of image processing pipelines in astronomy. Due to the volume and complexity of data that will be generated, modern astronomy is already undergoing a paradigm shift. New scientific megaprojects such as Vera Rubin, SKA, or E-ELT have created engineering challenges that have made it necessary to develop new techniques and models that take into account the nature of the data they will process. As deep learning architectures are known to function in a state where there are many more parameters than training examples, this requirement is directly related to the design of the model architecture. When considering the volume of astronomical data, this type of architecture has a high energy cost in several areas. The speeding up of convolutional neural networks can be achieved through a variety of techniques and approaches. Nevertheless, reducing computational power, memory, and energy consumption while maintaining model performance always entails a trade-off. An energy-efficient object detection model is presented here for morphological classification and galaxy location. In this presentation, we will present a number of techniques, including results demonstrating that tensor and running methods applied to the architecture's backbone yield a powerful model that approaches the results of the current SOTA. However, the most significant contribution is a 40% reduction in the number of parameters and a 30% reduction in energy consumption.

Since its launch in 2008, the Fermi-LAT Dataserver has been serving publicly available data to the Gamma-ray Astronomy community from NASA's Goddard Space Flight Center. In 2022 the legacy hardware running the LAT Dataserver was decommissioned and modern servers brought online as replacement. In this 15 minute contributed talk the Senior Engineer in charge of the transition will present the system's architecture, new software, performance improvements, design trade-offs, lessons learned, and future opportunities for the fully operational Fermi-LAT Dataserver.

Poster block 3

We present center-surround application to James Webb Space Telescope (JWST), Near Infrared Imager and Slitless Spectrograph (NIRISS) observations of Io, Jupiter’s innermost moon. This project is part of the JWST Early Release Science program (ERS1373) on the Jovian system. Io is the most geologically active body in our solar system however the locations of tidal heating in the interior and temperature of Io’s magma leading to the volcanic eruptions are not well constrained. NIRISS's Aperture Masking Interferometry (AMI) mode that utilizes a 7-hole non-redundant mask (NRM) in its pupil plane provides high-resolution imaging with moderate contrast, and better astrometric accuracy over a wide field of view than conventional imaging. We present interferometric observations of Io from ERS1373 using NRM and filter F430M that is well matched to the emission from Io’s 500 K to 2000 K lava flows. NIRISS' AMI mode is the first time NRM is used in space. Convolution of Io images with center-surround kernel is used to emphasize fine structure on Io's disk. This convolved image can then be used as a 'prior' to reconstruct images from interferometric data of Io, thus helping to provide new measurements of the global distribution of vulcanism on Io and improving constraints on the locations of tidal heating in the interior.

The B612 Asteroid Institute has developed the Asteroid Discovery, Analysis, and Mapping (ADAM) platform to analyze and understand asteroid data sets. ADAM uses Google Compute Engine to perform precision cloud-based asteroid orbit propagations, orbit determination, targeted deflections, Monte Carlo impact probability calculations, orbit visualizations, and asteroid discovery. Our vision with ADAM is to create a cloud-based astrodynamics platform available to the scientific community that provides a unified interface to multiple tools and enables large-scale studies. ADAM includes pre-configured settings to match common practices, such as the use of various time standards and coordinate frames, removing the need for users to perform any necessary transformations for comparison to results from external tools. ADAM's architecture consists of a web-service front-end, API interface, cloud-based storage, and cloud-based compute engines encapsulating multiple tools for computation and analysis.

We present the overall architecture and technical implementation of the ADAM platform, including planned and existing services. We go into greater depth about ADAM’s object precovery service, detailing the procedures used to normalize and index observation catalogs, the search algorithm, and the cloud computing infrastructure employed. We also describe the platform’s performance and present results of precovery searches both to extend the arcs of asteroids discovered by the Tracklet-less Heliocentric Orbit Recovery (THOR) algorithm, and to link orbits of the 1.2 million known minor planets with observations in the NOIRLab Source Catalog (DR2).

The detection of moving sources in astronomical data is the backbone of many planetary astronomy projects. To date, this task relies on costly visual vetting to confirm moving sources amongst the much more numerous stationary sources. This is especially true of surveys which search stacks of sequential images that have been shifted at rates of motion relevant to the bodies of interest. When the shift rate matches that of a moving source, a point source is revealed. This process provides a search depth that is comparable to the point-source depth that would be had from a single sidereal stack. As sources are not visible in individual frames, the so-called shift’n’stack technique comes at the cost of not being able to link detections as a source moves through the frames. This results in maximal human search cost, even after applying modern processing techniques such as image subtraction to remove most of the stationary chaff. Here we present a new machine learning technique to identify high probability candidate moving sources, geared specifically for shift’n’stack data. We make use of a multi-layer resnet to perform the binary classification task: good or not good? Our network is trained on artificial sources that were injected into the data before image subtraction, themselves incorporating a rate of motion matching the objects of interest. We have applied this network in a search for Kuiper Belt Objects (KBOs) in data acquired with the Hyper Suprime-cam on the Subaru telescope, as part of a search for targets for NASA’s New Horizons Kuiper Extended Mission. The network’s classification performance is extremely good, resulting in a reduction of spurious candidate sources by more than three orders of magnitude. An entire night’s worth of search data requires roughly only one hour of human vetting. We find a detection efficiency >80% for r<25.5, with a limiting magnitude of r~26.5-26.8 (depending on image quality) despite the fact that these data were acquired at a galactic latitude of ~10 deg (see Figure 1). A handful of the >200 newly detected KBOs are bright enough to be imaged directly by the Long Range Reconnaissance Imager on board the spacecraft, and are scheduled to be observed during 2022-25. Our results show a promising new avenue for moving object detection that has the potential to greatly increase the depth of upcoming large surveys such as the Vera Rubin Observatory Legacy Survey of Space and Time.

Figure 1: Example detection efficiency of 2022 Subaru/HSC data searches. The pre-ML vetted shift’n’stack sources are shown in blue, and possess a ~300:1 false positive rate. The post-ML results classified by the resnet is shown in orange, and the final human confirmed sources in green. A near-zero false positive rate is found after human+resnet vetting, down to the limiting magnitude (r~26.5).

While the availability of autonomous digital sky surveys has been revolutionizing the field of astronomy, they also introduce new challenges in the processing and analysis of the massive databases that they generate. One of the common approaches to the annotation of large databases of astronomical objects is by applying machine learning, and specifically artificial neural networks. Neural networks have demonstrated high efficacy in assigning correct annotations to astronomical objects, and can automate the analysis of databases that are far too large to be annotated manually. But while artificial neural networks can be an invaluable tool for astronomical data analysis, they also have several downsides. Here we study and profile the possible disadvantages of artificial neural networks in the context of astronomical data analysis. The study shows that when using artificial neural network algorithms, the annotation can have subtle but consistent biases. These biases are very difficult to detect, can change in different parts of the sky, and are not intuitive for the consumers of data products annotated by machine learning and deep neural networks. Since these catalogs are in many cases very large, these subtle biases can lead to statistically significant observations that are the result of the neural network bias rather than a true reflection of the Universe. Based on these observations, catalogs annotated by current artificial neural networks should be used cautiously, and statistical observations enabled by such catalogs should be analyzed in the light of possible biases in the machine learning systems. The results reinforce the need for further research on explainable neural network architectures applied to the field of astronomical data analysis.

Telescope archives are commonly perceived as static entities that store stale observations. In this talk we reimagine the humble telescope archive as a dynamic and resilient entity capable of responding to the rapidly evolving operational challenges facing observatories. Apart from the environmental spectres of climate change and pandemics, key challenges include coordinating multi-telescope observations of the imminent deluge of transients and facilitating equitable access to reduced science-ready data products. Addressing all these challenges requires the adoption of a sustainability mindset and embedding novel software components within the telescope archive. A notable example of the latter may include a Pipeline As a Web Service (PAWS) to enable on demand reduction of archival data, but other components interacting with transient brokers and telescope schedulers may be envisaged. We introduce a series of measures that could help transform telescope archives into active participants of a sustainable observatory ecosystem.

This talk presents the publicly available AstroML Interactive Book project.
Utilizing astronomy examples and datasets, the book introduces a variety of statistical and machine learning tools using popular Python libraries such as AstroML, Scikit-learn, and Scipy.
We also showcase the development process and the infrastructure, which enables the collaborative and interactive nature of the book.
The infrastructure itself is built on open-source libraries from the Jupyter and Executable Books ecosystems and facilitates a sustainable pathway for updating the existing material and extending it with new content.

Rust is a recent programming language that benefits from many appealing features and modern tooling.
The CDS began an evaluation of Rust in 2018, and it has now been introduced into several astronomical
tools and services such as the CDS HEALPix library, ExXmatch, MOCPy or Aladin Lite V3.
Based on the experience of using Rust at CDS we will discuss the advantages that we have gained
in terms of performance, stability and low memory footprint.
We will emphasise the benefits of the interoperability of Rust with other languages
and how this promotes the re-usability of code.
The examples of MOC LibRust and its derivatives (MOCCli, MOCPy and MOCWasm) are used to demonstrate
that these libraries can easily be called from a command line interface, through a Python wrapper or from a web browser.
With the development of WebAssembly, we describe how Rust could be used for both "code near data computations”
and to fulfil the old Java dream "Write Once, Run Everywhere".

The Liverpool Telescope (LT) provides robotic, autonomous observations for the time-domain community with rapidly reduced data available to users minutes after observations are taken. An intelligent dispatch scheduler, developed in the early 2000s, ranks observations according to various factors such as observational constraints, position on the sky, weather and scientific priority. However, there is a requirement to exploit advancements in Artificial Intelligence to optimise robotic follow-up strategy and meet the demands of our ever-growing survey telescope domain with an intelligent scheduling solution.

The New Robotic Telescope (NRT), a 4-metre facility designed for rapid classification of transients, combines mechanical, Artificial Intelligence and hardware advances to create an extremely rapid response facility, slewing across the sky and acquiring data within 30 seconds of trigger receipt. The scheduler is crucial in streamlining the follow-up process. We can exploit the vast LT database of observations to develop optimal scheduling algorithms before the NRT’s first light in 2026.

Each LT observation carries many associated data in FITS headers and ancillary data, such as weather reports and duty officer logs. These are often stored in a distributed manner where relationships within the data are time-consuming to extract. In this project, we have used Microsoft Power BI to create relational links between these data: over 594 unique FITS headers from approximately 4.1 million observations. We can then query the relational tables in near real-time to generate subsets of these data where valuable insights into the operation of the LT can be gained. These subsets of data can be used to train several different algorithms to uncover hidden trends and facilitate real-time intelligent scheduling.

In this talk, I will present the methodology and early findings from this project and discuss how these insights will help to inform future work in building optimal scheduling algorithms for the New Robotic Telescope.

For well over a decade now, LOFAR ran on an aging collection of tools to specify and manage its observations and processing pipelines. These tools were written in several different languages and technologies. Each had its own model of the telescope, forcing us to do a lot of translations and communication between them. This setup greatly hampered extending the telescope model with new features, and carries an operational cost that cannot easily be lowered. In short, we were locked in.

With TMSS, the Telescope Manager & Specification System, we upgraded LOFAR's main interface to specify and manage observations and pipelines. We setup a modern Python Django + React stack, on top of a telescope model in PostgreSQL.

In this talk, we show how we combine relational and NoSQL constructs to create a robust yet extendable model for our telescope throughout the full stack, using JSON Schemas to validate specifications and even generate GUIs. How we added a high-level features such as a QA workflow between user and system, and a run-time (dynamic) scheduler of observations, based on their constraints. And finally, we touch on some of the lessons learned during development and deployment into production.

MOONS is an optical near-infrared multi-fiber spectrograph to be
mounted on the ESO VLT. Its Field of View of 25 arcmin in diameter
is covered by ~1000 robotic Fiber Positioner Units. To fully exploit
such a high multiplexing, while accounting for hardware constraints,
the observing conditions and setup, and the user preferences is a
challenge. While the astronomer can assign priorities to targets and
tune it, the process of fiber allocation must be automated. By defining
a figure of merit and isolating the hardware-specific features, this task
reduces to a complex discrete optimisation problem. I will describe the
algorithms and the logic behind the MOONS target allocation
software which will be offered with the instrument.

The CHaracterising ExOPlanet Satellite (CHEOPS) is ESA's first small class mission and is a partnership between Switzerland and the ESA Science Programme, with national contributions from Austria, Belgium, France, Germany, Hungary, Italy, Portugal, Spain, Sweden, and the United Kingdom. Launched on December 18th, 2019, with a mission lifetime of 3.5 years, CHEOPS is a follow-up mission that measures the radii of exoplanets known to transit bright host stars (Vmag 6-12). Together with the mass measurements from other instruments like ESPRESSO at ESO's Paranal site, CHEOPS provides well suitable targets for spectroscopic follow-ups with space- or ground-based observatories like the JWST or the E-ELT.

The CHEOPS Science Operations Centre (SOC) is responsible for the mission planning, the data reduction and the dissemination of science-ready data products through the CHEOPS mission archive. The small class nature of the mission places strict requirements on the SOC; the short development time and limited man power requires a high level of automation and the re-use of existing hardware and software resources. We present here the high-level architecture of the SOC software system and describe challenges and lessons learned from developing and operating the SOC of a small mission.

At the European Space Astronomy Centre (ESAC) near Madrid, the ESAC Science Data Centre (ESDC) hosts ESA archives for Astronomy, Planetary and Heliophysics Space Science. Furthermore, the GNSS Science Support Centre (GSSC), with special attention to Galileo and EGNOS, consolidates an ESA archive for scientific exploitation of Global Navigation Satellite Systems (GNSS). The deluge of data generated by ESA missions in Space Science, Navigation, Earth Observation, and other domains both, from a scientific and operational viewpoint, calls for a brand-new palette of capabilities able to extract insights from these multi-mission, federated data sources.

Built around science-return as its central pillar, ESA Datalabs aims to present itself to the end-user as an intuitive system for swift access to a large catalogue of data volumes and processing tools effectively integrated in a single platform. Behind this glossy curtain, hidden from the user, lays an IT infrastructure which features an extremely sophisticated architectural blueprint. Kubernetes clusters, Rancher, ElasticSearch engines, Docker containers, and many other usual suspects of the High-Performance Computing world, team-up to deliver an innovative experience.

This focus demo will guide the audience through multiple catalogues that shape the application store and software as service concepts present in ESA Datalabs. First, the Datalabs catalogue will be introduced. This catalogue provides access to data analysis tools ranging from domain-specific desktop tools like Topcat, well known astronomical tool, to general-purpose, web applications like JupyterLab, widely adopted for data science in multiple fields. Through this catalogue users can search, comment, bookmark, and run any Datalab. Once the user finds a Datalab of interest, a simple click on a play icon makes the magic. At this point, users can modify the default configuration of the Datalab set by its creator, select a previous version of the Datalab, increase its computing resources, or connect additional data volumes from ESAC Archives. Following Datalab start-up, users can go back to the catalogue and launch up to five Datalabs in parallel (default profile configuration). Furthermore, a wizard-like editor guides ESA and non-ESA developers to make Datalab contributions following a build and moderation process that includes automatic security scans.

Leveraging on the powerful infrastructure developed for the execution of Datalabs, this demo will introduce a second catalogue for the execution of complex, batch data processing Pipelines. Pipelines represent an extension of the Datalab entity, defined as a set of an input area, a sequence of processing stages in between (steps), and an output area. The capabilities to perform data integration, pre-processing, transformation and analytics represent the entry point for Machine Learning Pipelines into ESA Datalabs. Moreover, a visual editor provides an integrated development environment to put together these processing workflows in a graphical way. The editor drives the user through the development cycle, simplifying the creation process and transforming the graphical representation of Pipelines into Common Workflow Language (CWL), the underlying standard supported by the orchestration engine.

Throughout this focus demo, the audience will see how ESA Datalabs catalogues permeate through the Data Archives at ESAC, bringing new collaboration features derived from the possibility to share co-located computing elements and storage areas. Along these lines, several JupyterLabs will illustrate how to explore and analyse data from ESAC archives.

Currently available as a beta release, ESA Datalabs joins the growing number of science exploitation platforms, implementing advanced network and computing capabilities to leave behind the discovery and download science era, showcasing a new era characterized by archives tight coupling with exploitation tools as a service.

SourceXtractor++ is an open source software for detecting and measuring sources in astronomical images. It is a complete redesign of the original SExtractor2, written mainly in C++ and to serve the needs of the modern astronomy community. The package is written following a modular approach and facilitates simultaneous source analysis over many images with different pixel grids. The flexible model-fitting of SourceXtractor++ is configurable via a python interface in order to solve the diverse astrophysical problems in the multi-channel and multi-epoch domain.
After an initial alpha release for ADASS XIX we have now implemented all anticipated features and also achieved full compatibility with SExtractor2. The software package has excelled in the Euclid Morphology Challenge, a recent comparison of several up-to-date model fitting solutions. In this demo we show a classical application of SourceXtractor++ as well as applications demonstrating the extensibility of SourceXtractor++ with user provided fitting models. We show how to use ML models in SourceXtractor++ for source detection and source properties via inference. SourceXtractor++ is being developed in the context of the Euclid satellite projects, however it is distributed as an independent package via source code or various binary distributions.

The majority of the time spent on astronomical data systems is on practical matters which is never published, from recruitment to legacy systems. This BoF is a forum for these common problems. The discussion (which will not be recorded!) will be based on these three questions:

• What are your biggest problems right now, and concerns for the near future?
• What would you do differently at the start of your current project?
• What experiences would you like to share?

Most ADASS participants use a Linux flavor and/or MacOSX. I'm going to
need some crowdsourcing type help on this, but I would like a
community writeup of the tools of our trade, anno 2022. During COVID I started a
writeup for Ubuntu Linux, since that's what I'm using, based on a now
well aged writeup for a Mac. You can see this draft on

https://github.com/teuben/teunix/blob/master/docs/astronomy-on-ubuntu.md

This is geared towards an audience who already have their fingers wet
on linux, but may not know all the tricks of the trade to get a full
astronomy toolbox.

I will need one or two more co-conspiritors.

Space missions produce an unimaginable amount of data, which at some point has to be: cleaned, processed, transformed, and passed through pipelines. Later on, the data will be in one way or another stored and analysed. Multiplying that amount by the number of missions shows that not only the tools, but also the architecture, should support the immense volume of information.

These sets of mission data appear perfect to use for data analysis, with the application of various libraries or algorithms, but a question is introduced - how should the size be mitigated, so that the time complexity of the operations does not fall into the worst-case scenarios? One way to accelerate computational operations on big data is with Graphical Processing Units, which utilise the idea of SIMD - a single operation repeated on multiple data points. While the user will see the immediate benefit in the speed at which the result is obtained, behind the scenes, this means an effective use of threads, memory, and concurrent access to resources.

The other crucial aspect to be considered is sharing the data with the community. Most of the times moving or copying it across the Internet is both complex and time-consuming, so a good solution would be to bring the user to the data. This is in essence how ESA Datalabs works - the platform brings the user and their code to the information and offers custom tools for handling astronomical data, as well as those of a more general purpose (like Jupyter Notebooks). ESA Datalabs is built on Kubernetes clusters. This approach allows independence from a particular operating system with minimal virtualisation overhead. Managed properly, the clusters offer persistence and most importantly, scalability - if the users need more resources or the platform has to scale, this can be handled by adding new clusters, for example, one with a GPU.

This presentation is going to introduce the concept of GPU computing and multiclustering, how a single-cluster architecture can be expanded, and the considerations to be taken when integrating GPUs into these big data processing environments.

The technique of interferometric imaging has enabled remarkable
precision, resolution and sensitivity of telescopes at radio
wavelengths. Sensitivity of an Interferometric telescope scales with
the number of antennas in an antenna-array telescope. While the
resolution is proportional to the largest separation between these
antennas, the precision depends on the stability of the signal
collected over a range of time and frequency. Noise limited imaging
with such telescopes therefore critically depends on precise
calibration of the data, which in-turn requires detailed understanding
of the various electronic and atmospheric elements in the signal path.
With the wide-band receivers of modern telescopes, wide-field
wide-band imaging using calibrated data requires detailed
understanding of the optics involved in using the technique of Earth
rotation aperture synthesis. Such telescopes are fundamentally
indirect imaging devices which only partially samples the data space.
Noise-limited image reconstruction is therefore fundamentally an
iterative process. Along with large volume of data required to reach
the full potential of the telescopes, this leads to enormous
size-of-computing requirements. The need to understand telescope
hardware, instantaneous atmospheric models, telescope optics, signal
processing techniques, computational numerical techniques, computer
science techniques, scientific software development techniques and
High Performance Computing (HPC) and High Throughput Computing (HTG),
makes algorithms R&D a highly inter-disciplinary field of research.
Depending on the individual interest, this makes research in this
field exciting and intellectually challenging with opportunities
for continuous growth.

In this talk, with the goal of conveying the excitement, joys and
challenges of algorithms R&D in this area, I will give an overview of
what I think is required in terms of calibration and imaging
algorithms to enable the full potential of a modern Interferometric
radio telescope in a manner that allows users of the telescope to
largely focus on the astrophysical goals. This will include experience
of my personal journey in this field, fundamental advances in the past
few decades, the current state-of-the-art algorithms in a few areas of
interest and some results. I will also spend some time discussing
what I think are the current and future needs -- specifically to
convey the challenges and the excitement that lies ahead.

As the Very Large Array Sky Survey (VLASS) observes the entire Northern sky in radio, it generates quick-look images. These images often show residual artifacts, particularly around brighter sources. Most of these artifacts in VLASS are linear structures due to the VLA’s antennas mainly stretching in three lines and the snapshot nature of observations (VLASS often only has less than 5 seconds of data at a given location.) While well-established techniques (like CLEAN) maximize the information from snapshot imaging, these techniques can be imperfect. This incomplete image reconstruction is particularly true for surveys that prioritize the speed with which they generate large numbers of images to capture variable radio emission on fast timescales. Moreover, as surveys like VLASS become increasingly common, automatic image-quality classification is increasingly important for rapid data quality assessment and enabling the best science. For example, knowing whether a part of an image is affected by the above-mentioned linear streaks is critical for distinguishing truly variable radio sources from artifacts, among other science goals. Here I present new methods (e.g., prediction from the Fourier transform of the uv-sampling and empirical line detection using the Hough transform) for identifying residual linear structures in VLASS images. I also discuss the potential application of these methods to both on-the-fly and pointed observations with the VLA outside of VLASS and to observations with other interferometric arrays.

Poster block 4

The James Webb Space Telescope (JWST) is currently the major space science observatory after a successful launch and commissioning phase. The ESAC Science Data Centre (ESDC) has been in charge of providing a European JWST Science Archive (eJWST) with the aim of boosting the science return of this great observatory. Within the established partnership of NASA, ESA and CSA, all JWST metadata and public data is being synchronised in near real time mirroring the Mikulski Archive for Space Telescopes (MAST) JWST archive at the Space Telescope Science Institute (STScI). The ESDC provides a carefully designed graphical user interface plus several backend services based on Virtual Observatory (VO) protocols to access both private (data stored at STScI) and public observations (data also stored at ESDC).

In this demo we will show how eJWST enables a quick, straightforward and user-friendly access to Webb’s data. We will explore the sky putting JWST observations data and metadata in the context of multi-wavelength science with the integration of the ESASky tool. Quick preview of data is also possible by means of archive data viewers.

We will also illustrate how the eJWST Archive provides several means to accelerate science by providing users access with searches based on the ADQL query language and dedicated python modules in packages such as an eJWST-specific Astroquery module. This package is included in the JWST area in ESA Datalabs, the new science exploitation platform developed by the Data Science and Archives Division at ESAC. A tour of the analysis, on-line processing and collaborative research that this platform offers will also be demonstrated.

The sample of Solar system objects has dramatically increased over the last decades. The amount of measured properties (e.g., diameter, taxonomy, rotation period, thermal inertia) has grown even faster. The benefit of all these developments has, however, not come to full fruition. While some catalogs are publicly available in machine-readable formats on, e.g., the Planetary Data System (PDS), or the Centre de Données astronomiques de Strasbourg (CDS), a significant fraction of results are only tabulated within articles. Furthermore, the designation of small bodies often evolves with time, from potentially several provisional designations, to a single number and finally an official name. Hence, the same object can be referred to by different labels in different studies, making its cross-identification over several sources a complex task. Accessing to all the characteristics of a given body, or a population, can thus be tedious, or even impractical. A universal access point for all measured properties of Solar system objects available in the literature and online databases is thus required.

We provide a practical solution to the identification of Solar system objects from any of their multiple name/designation. We also compile and rationalize their properties to provide an easy access to them. We aim to continuously update the database as new measurements become available. We built a Web Service, SsODNet, that offers four interfaces, each corresponding to an identified typical need in the community: name resolution (quaero), compilation of a large corpus of properties (datacloud), determination of the best estimates among compiled values (ssoCard), and statistical description of the population (BFT).

The name resolver quaero translates any of the 5 million designations of objects into their current official designation in a fraction of second. The datacloud compiles about a 100 million measurements of many parameters (osculating and proper element, pair and family membership, diameter, albedo, mass, density, spin, phase function, color taxonomy, thermal inertia, and Yarkovsky drift) from almost 3,000 articles. Each parameter is associated with an object, and a complete bibliographic reference is linked with it. For each of the 1.2 million objects in the system, a ssoCard providing a single best-estimate for each parameter is available, and delivered in json format. Finally, the BFT large table compiles all these best-estimates into a single eCSV file for population-wide studies.

The four interfaces of SsODNet have a Web service with an application programming interface (API). The SsODNet interfaces are fully operational and freely accessible to everyone: https://ssp.imcce.fr/webservices/ssodnet/ .
We also proposes a python package and command-line tool to acces the service: rocks (https://rocks.readthedocs.io/).
We will present the motivation behind the service and a live demo of how to use it and its performances

Additionally to the development of new, modern software, the support
of legacy code remains an important problem in the astronomical
community. Data reduction and analysis pipelines may depend on old
software tools, without strong motivation and funding to break this
dependency. Also, despite of strong efforts, modern software tools
still may miss some features that are provided by old software. On the
other hand, legacy software often lacks institutional support and also
does not follow current software development principles.

I will discuss the motivation and experiences for the support of
legacy software, and the specific challenges for IRAF maintainance as
an example.

SIMBAD is a service that provides reference data about celestial objects for astronomers that has been in operation for more than 40 years. Over this time astronomical data themselves have evolved because of the technological evolution of the observations, but also because of the evolution of software engineering. SIMBAD has gone through four major versions, each of which has been built using a different programming language, or other architecture, but with a long-term view to be scalable for future evolutions.

For instance, the very first interactive version of SIMBAD, in 1981, allowed queries for data with a constraint on the sky coordinates. Now, in the 4th version of SIMBAD, this service is still available, along with many other options. For each new version, in order to assure back-compatibility, the URLs and interfaces already used in the past are still available in parallel to the new version. The maintenance of such legacy software is necessary for the continuity of the service and to avoid problems for the many other community services that use SIMBAD. The support of such a software is challenging because it cannot be fully integrated with new developments. We will describe the range of different solutions that we use: specific redirections, hacks, interfaces, filters that are currently used for SIMBAD utilities.

Brown dwarfs are intermediate objects between stars and planets. Their mass is not enough to start and maintain stable hydrogen fusion, which causes them to cool over time. Exploration of brown dwarfs is interesting for several reasons. First, the transition boundary between planets and brown dwarfs is not fully understood. Second, atmospheric properties of brown dwarfs strongly affect their photometry, which can not be fully explain yet by modern atmospherical models.
Homogeneous and complete samples of brown dwarfs are needed for these kinds of studies. Due to their weakness, spectral studies of brown dwarfs
are rather laborious. For this reason, creating a significant
reliable sample of brown dwarfs, confirmed by spectroscopic
observations, seems unattainable by now. Numerous attempts have been made to search for and create a set of brown dwarfs using their colors as a decision rule applied to a vast amount of surveys' data. In this work we use Random Forest Classifier, XGBoost, SVM Classifier and TabNet on PanStarrs DR1, 2MASS and WISE data to distinguish L and T type of brown dwarfs from objects of other spectral and luminosity classes. We also compare our models with classical decision rule models, proving their efficiency and relevance.

The ADASS Program Organizing Committee is pleased to announce the winner of the annual ADASS Prize for an Outstanding Contribution
to Astronomical Software.

The 2022 recipient of the prize is the Astropy Project. Since its formal beginning in the Fall of 2011, Astropy Project has been a community effort with the combination of institutional resources and many dedicated contributors. It has grown over the years and been used by many projects and individual astronomers. The first core astropy package release was version 0.2 in 2013. It is now at version 5.1. It is known to almost everyone involved in astronomy software development.

This award is presented in recognition of the outstanding contribution of Astropy Project to the astronomical software community and the positive impact it has on many astronomy projects. The Astropy Project is made possible through the hard work of hundreds of people in the community. Please see https://www.astropy.org/team.html and https://www.astropy.org/credits.html for details.

The ESAC Science Data Centre (ESDC) is handling the archive data for several astronomy, solar and planetary missions. We started with some gigabytes of information, currently in the hundreds of terabytes and not so far in the future we will handle with petabytes. Some important fraction of them reside in database systems which allows to analyse the data directly in the ESDC systems using VO protocols. All kinds of data: structured, semi and unstructured. How to store this data in a database to give the users the ability to query easily the contents of a space mission?. How do we choose a solution that will handle some small data to one that scales better for big data. May this be a nightmare?

One does not fit all, but maybe in the future it well may happen.

We will review the evolution of database solutions for big data space projects with special focus in the ones that we have already tested (PostgreSQL, CitusDB, PostgresXL, Greenplum) with specific implementation for the Gaia DR3 release, the European JWST archive, the Euclid Science Archive and the future PLATO Data Archive.

We present recent developments in the open-source "Firefly" astronomical data access, exploratory analysis, and visualization toolkit. Firefly is the core library from which the NASA IRSA archive web interfaces and the Rubin Science Platform's Portal Aspect are constructed, and is also used in many roles in the NASA Extragalactic Database (NED) and Exoplanet Archive web interfaces.

Firefly is evolving rapidly into a general-purpose tool for exploring astronomical data through IVOA standard data-access protocols. Recent developments have extended Firefly's capabilities in querying observation data in the ObsCore data model, via ObsTAP, and this has allowed the deployment of image-query services for the Rubin Data Preview exercises in a purely standards-based way.

We will also describe extensive new capabilities built around the IVOA DataLink standard. These have allowed providing connections between related datasets, such as the retrieval of light curves or of the image on which a source was observed, in a completely standards-based way that avoids the need for putting mission-specific information into the data-access application. DataLink also allows us to construct query screens for datasets in an entirely metadata-driven way, allowing new query capabilities to be introduced without changes to the front-end application.

We will also present recent enhancements to Firefly that extend its capabilities as an exploratory-data-analysis tool for astronomical images and catalogs.

Firefly is open-source software and a Firefly server is available as a pre-built containerized application, with substantial run-time configurability. We will show how to use the container image to rapidly establish an archive query interface for standards-compliant data services.

Since its first version in 2013, Aladin Lite has gained significant traction and usage as an HiPS viewer running in the browser. Designed to be easy to embed, it is now used in more than fifty websites and portals in the professional astronomy community. Aladin Lite has been adopted as the sky visualisation component of popular applications: ESASky, ESO Science Archive or ALMA Science Archive and has been supported in the ESCAPE project as a priority for Astronomy infrastructures.

We present a major overhaul of Aladin Lite taking advantage of the GPU with WebGL, and which responds to requests of users, developers and integrators in a context where browser-based applications and science analysis platforms are increasingly important. While keeping the strengths of the original code including its API, Aladin Lite v3 introduces several new features: support of multiple projections (Aitoff, Mollweide, Orthographic, Mercator), support of FITS HiPS tiles, an improved rendering pipeline and coordinates grids. In addition to video footage sequences presenting those new features, this talk will be more focused on how users can embed Aladin Lite v3 into their own applications. This presentation also aims at Aladin Lite v2 users to upgrade to the v3 with very minimal changes to their codes.

Finally, a point will be addressed to future developments including the support of FITS images as well as a better MOC IVOA standard support.

These improvements will also benefit ipyaladin, the widget enabling the usage of Aladin Lite in Jupyter notebooks.

The Julia programming language can be considered the successor to Scientific Python (SciPy). The language is designed for scientific computing by having built-in multi-dimensional arrays and parallel processing features. Yet, it can also be used as a general-purpose programming language like Python. Unlike Python, Julia solves the two-language problem by using just-in-time (JIT) compilation to generate machine code from high level expressions. In most cases, Julia is as fast as C, and in some cases faster. Julia is also a composable language, so independent libraries or packages usually work well together without any modification. These important features make Julia a very productive language for scientific software development by reducing the number of lines of code.

The Data Activate Flow Graph Engine (DALiuGE) is a workflow scheduling and execution system. It has been developed with extreme scalability ability, ranging from stand-alone laptops to the biggest supercomputers in the world, as one of the key requirements. One of the other requirements was to be able to re-use existing (radio astronomy) software, while also allowing a tighter integration of dedicated processing components with the execution engine. The third main requirement was separation of concerns to allow astronomers and workflow developers to concentrate on the workflow logic and the selection of appropriate algorithmic components, software engineers on the development and maintenance of operational grade components providing those algorithms, HPC specialists on the optimization of the execution of the components and the overall workflow and finally also enable hardware/software co-design for performance or I/O critical components. In the meantime DALiuGE has reached beta status and is ready for broader consumption. An earlier version has been used to drive the 2020 Gordon Bell prize finalist project to demonstrate that we would be able to process SKA scale data streams on a supercomputer the size of Summit. During that run we have used 99 percent of Summit (27,360 GPUs), achieving 130 petaflops peak performance for single-precision, 247 gigabytes per second data generation rate, and 925 gigabytes per second pure I/O rate.

This paper presents an overview of the system and cover the key aspects of the implementation of each of the requirements mentioned above.

Spectroscopic redshifts of quasars are important inputs for constructing many cosmological models. Redshift measurement is generally considered to be a straightforward task performed by automatic pipelines based on template matching.

Due to the millions of spectra delivered by surveys of SDSS or LAMOST telescopes, it is impossible to verify all redshift measurements of automatic pipelines by a human visual inspection. However, the pipeline results are still taken as the "ground truth" for further statistical inferences.

Nevertheless, because of the similarity of patterns of quasar emission lines in different spectral ranges, an optimal match may be found for a completely different template position, causing severe errors in the measured redshift. For example, it may easily happen that a faint emission star with a noisy spectrum is identified as a high redshift quasar and vice versa.

We show such examples discovered by the consistency check of redshift measurements of the SDSS pipeline and redshift predictions of a regression Bayesian convolutional network. The network is trained on a large amount of human-inspected redshifts and predicts redshifts together with their predictive uncertainties. Therefore, it can also identify cases where predictions are uncertain and thus require human visual inspection.

Closing remarks (LOC Chair), and an announcement for ADASS 2023