"We've only tested the happy path - now users are finding all sorts of creative ways to break the app."

What is already a cause for headaches in traditional software engineering turns into a large challenge when the application is based on machine learning models: Data distribution may change from training phase to deployment. Even worse, humans interacting with the model may adjust their behaviour to the model making the gap between original training environment and deployment even larger. When deployed in a public environment the model may be exposed to users trying to game the system. When re-trained it may be exposed to users trying to poison the pool of training data.

We will take a tour of historic cases of models being gamed: What are the lessons we learnt a long time ago building e-mail spam filters? What happened when high search engine rankings started to be linked to monetary income? How can personalization and targeted advertising be exploited to influence public discourse?

“… it should be clear that improvements in communication tend to divide mankind …” by Harold Innis in Changing Concepts of Time

This keynote will turn interactive engaging the audience in sharing their stories on users playing interesting games with deployed models - including counter moves rolled out.

If we are to learn from IT security experience, one important ingredient to address these issues is a combination of collaboration and transparency - across organisations.