2024-11-11 –, Aula Magna
The advent of supercomputing has the potential to revolutionise astronomical data processing and is essential for the analysis of massive Radio Astronomy datasets of the SKA era at unprecedented speeds. This talk presents our experiences implementing a complex data reduction pipeline on a number of state-of-the-art supercomputer facilities. We demonstrate how, when optimised, our pipeline achieves remarkable throughput, reducing processing times from weeks to mere hours for large-scale surveys.
However, the transition from traditional computing environments to supercomputing infrastructures is not without challenges. We discuss several critical issues encountered, including:
1) Scratch file management in distributed file systems like Lustre
2) File handle limitations in massively parallel operations
3) Scheduler conflicts and queue optimisation with measurement sets being very different in size depending on observation time and flagging
4) Wall time constraints and job segmentation strategies
We offer some solutions for astronomers looking to leverage high-performance computing resources and tools, such as DALiuGE, to mitigate many of these issues. Our findings highlight both the transformative potential and the practical considerations of supercomputing for modern astronomical research.
Kevin Vinsen is a Senior Research Fellow at the International Centre for Radio Astronomy Research (ICRAR) at the University of Western Australia (UWA). His work focuses on data-intensive astronomy, with an emphasis on developing machine learning methods for analysing large astronomical datasets.
Vinsen's research interests include:
1) Applying machine learning to process astronomical data
2) Exploring methods in data-intensive astronomy
3) Modelling complex astrophysical systems
4) Translating astronomical technologies into industry applications
His current work contributes to preparations for next-generation radio telescopes, including the Square Kilometre Array (SKA). Vinsen collaborates with international research teams and participates in global conferences, bringing an interdisciplinary approach that connects astronomy, computer science, and industry.