2024-11-11 –, Aula Magna
In 2021, the W. M. Keck Observatory and the NASA Exoplanet Science Institute enhanced their collaboration through the Data Services Initiative aiming to produce and archive science-ready data. This endeavor focused on refining two pivotal aspects of observatory operations: the planning and execution of observations, and the processes of data reduction and archiving. To achieve these, the team developed new tools, including web-based GUIs, to capture observer intentions into a machine-readable format and an execution engine to carry out these observation plans. A revamped data handling architecture was also introduced to expedite the creation of reduced data products and their subsequent near-real time ingestion into the Keck Observatory Archive. As a result, raw and reduced data are now made available to users well under the requirement of 5 minutes, significantly enhancing the efficiency of data access. Additionally, the number of community-supported data reduction pipelines for Observatory instrumentation expanded from 2 to 9 (across 11 instruments), exceeding the initial project goals. Despite these advancements, the team navigated significant challenges, such as integrating automated workflows within a classically scheduled observing model and managing with limited personnel and budgetary resources. This presentation will explore the project journey, highlighting the successes achieved, obstacles faced, and lessons learned throughout the development cycle that can be applied to your future work.