While astronomy data reduction and analysis software faces significant challenges, it has also seen major successes in standardisation, automation and collaboration, ensuring that data are processed efficiently and in ways that are accessible to ever larger fractions of the community.
The future of this field lies in ever greater automation, the incorporation of machine learning techniques for near-real-time data analysis, and more seamless integration of heterogeneous datasets. As the volume of data continues to grow, our challenge is to ensure that pipelines remain scalable, robust, and flexible enough to handle both routine and unusual datasets. Of course advances in algorithms and technology are only part of the solution: collaboration across disciplines - software engineering, astronomy, data engineering and computer science - is key to the success of this field.
In this presentation I will highlight some of the successes, and a few failures, of this field and explore how as a community we are preparing to tackle the challenges of the next generation of projects.