2024-10-26 –, CLASS #5 - 3B
Language: English
With all the hype about applications that uses machine learning, I think there is one key aspect that developers tend to forget: "Performance check and monitoring".
ML and AI services have become very accessible and can be integrated into any application you can think of. But what do you do after you integrated your ML models to your application? How do you know that the output of the ML models are correct and up to standard? What are the signs that the model's performance is changing and how do you take action on such changes?
Typically, these problems are just discussed on a theoretical and research level. But how can we carry over these techniques and apply it to our application? Not just that, how can we make it so that monitoring and performance check is as simple as writing a unit test (or not).
In this session, we will learn some simple but effective ways on model performance monitoring as well as look at some python implementation and architecture consideration. We will also check some best practices and some real life scenario on how model monitoring works.
The inspiration for this talk came from more than a year ago when my partner (software engineer) started looking into ChatGPT for her work. Almost every time, our conversation always ends with her asking "how do I know that the output of our application using chatgpt is correct?"
Model validation and performance checking is a very common topic on research and in the academe. However, due to the rise of AI as a service, more and more people started encountering this problem.
Having implemented machine learning applications for more than 7 years, model validation can be quite cumbersome, especially when dealing with high traffic and variable type of datasets. But through the years, we've picked up some simple but practical approach to add these features in our python applications.
The talk will focus on the following key ideas:
- Why do you need model validation?
- How do you choose the right validation metric for your ML application?
- Implementing a simple model validation and monitoring module in python.
- Understanding the model validation result and how this affects your ML application.
This session will require some basic understanding of machine learning but all the resource will be beginner friendly and code examples will be available for participants to try out in their free time.
