PRESC is a tool to help data scientists, developers, academics and activists evaluate the performance of machine learning classification models, specifically in areas which tend to be underexplored, such as generalizability and bias. Our current focus on misclassifications, robustness and stability will help facilitate the inclusion of bias and fairness analyses on the performance reports so that these can be taken into account when crafting or choosing between models.
This is a project sprint from the "AI IRL Hackathon - Building Trustworthy AI". Registration and more information here: http://mzl.la/taihackathon
PRESC is still a young project, and would benefit greatly from having its infrastructure and approaches to model evaluation tested and validated in a broader context. This session will be a hackathon sprint to contribute together to this project. We invite you to contribute by:
- Test driving the tool on your dataset and model
- Contributing a dataset for us to use for future testing and development
- Making code contributions
- Providing your perspective, feedback, or recommendations based on your experience or industry
As the tool is currently accessible as a Python library API, some experience with Python and its data science stack (Pandas/Numpy/Scikit-learn) is necessary to run it. Aside from this, participants are welcome to interact with the project through the Github repo, such by commenting on issues. High-level documentation on the evaluation approaches is also available in the repo for discussion.
We're hoping that many efforts and discussions will continue after Mozfest. Share any ideas you already have for how to continue the work from your session.:PRESC is an open source project, so contributions are always encouraged both before and after Mozfest.
How will you deal with varying numbers of participants in your session?:More participants will allow for a richer discussion, they can be separated in smaller work groups dealing with particular problems, and a lower number of participants will allow for a more personalized experience and discussions.
Researcher. Developer. Data Scientist. Human rights & freedoms activist, including their expression in a digital and technological context.
