Leonardo Amorim
Senior Data Scientist based in Tallinn, Estonia, with 9+ years of experience in machine learning, analytics, experimentation, and causal inference.
Session
Large scale transactional platforms use machine learning models to block fraud before it occurs. However, once a model changes which transactions are allowed, it also changes which labels can be observed. The model cannot see outcomes of transactions it considered risky. Metrics from the surviving population become biased, and retraining on those labels can reinforce the same blind spots.
A randomized holdout restores an unbiased sample of the affected population, but creates a second problem: how should a small holdout be combined with the larger censored dataset? This talk compares six retraining strategies in a controlled simulation built with LightGBM. It shows when simply dropping censored labels is preferable, when inverse propensity weighting (IPW) recovers hidden information, and why neither method dominates across data regimes.
We then introduce Asymmetric IPW, which uses pooled holdout validation to choose between two retraining methods: dropping and pure IPW. Across six scenarios, it tracks the stronger method without requiring the underlying regime to be diagnosed. Attendees will learn why a holdout is necessary for unbiased evaluation, how censoring affects retraining, and how validation can balance bias and variance when retraining a model after launch.