2026-09-10 –, Room 1 (170)
Large scale transactional platforms processing millions of daily events require high-performance algorithmic fraud intervention. However, deploying models that block bad actors introduces a structural blind spot: the censorship of positive labels needed to train the next generation of models. To generate unbiased data, systems typically rely on a holdout group, a subsample where the model doesn't act. This creates an operational tradeoff: a large holdout caps the model's ability to intervene, while a small holdout reduces the sample size of the training pipeline, especially when the labels are heavily unbalanced.
This talk details the methodology developed to mitigate this reject inference using LightGBM and Optuna. We will walk through the mathematical theory of Inverse Propensity Weighting (IPW) to reconstruct missing data, and examine why applying standard IPW directly to GBDTs often causes gradient explosion and overfitting. To bridge the gap between statistical theory and tree-based mechanics, we will introduce Asymmetrical IPW: a hybrid weighting architecture that stabilizes tree gradients using optimizable, asymmetrical weights. This approach yielded a model recovering 99% of the benchmark's partial AUC (@20% FPR), where the benchmark is a model trained on fully uncensored data.
Topic and Relevance
Reject inference is a common problem in fraud detection, credit scoring, and algorithmic moderation, yet standard literature often focuses on linear models. This talk specifically addresses the clash between statistical dataset reconstruction (IPW) and the non-linear, recursive partitioning mechanics of Gradient Boosted Decision Trees (GBDTs). We will share the learned lessons on why theoretical fixes often fail in production and how to search for custom loss weights that actually work.
Target Audience
Data Scientists and Machine Learning Engineers working in fraud, risk, or any domain suffering from censorship/survivor bias.
Audience Takeaways
- Understanding the mechanics of censorship bias in production fraud models and the operational limitations of reliance on holdout groups.
- Identifying the mathematical reasons why standard Inverse Propensity Weighting (IPW) causes gradient explosion and overfitting when applied to Gradient Boosted Decision Trees (GBDTs).
- Applying Asymmetrical IPW methodologies to stabilize tree gradients and optimize custom loss weights for highly imbalanced datasets.
Talk Type and Approach
This talk is conceptual and mathematical. It focuses on the theoretical conflict between statistical dataset reconstruction and GBDT recursive partitioning mechanics. The presentation will provide high-level architectural guidelines for implementing Asymmetrical IPW, rather than line-by-line code walkthroughs.
Required Background Knowledge
Attendees should have a working knowledge of Gradient Boosted Trees (LightGBM/XGBoost), handling class imbalance, and basic hyperparameter optimization concepts.
Scope and Structure (estimated 30 minutes total):
- Introduction, Context & The Censorship Problem (5 minutes)
- Fraud model with 1:99 class imbalance.
- The censorship problem and its effects on model performance.
- Creation of a 5% rotational holdout to mitigate the issue.
- The tradeoff of holdout size.
- Why holdout data is usually not enough to maintain a model's benchmark performance over time.
- Strategies to retrain the model with little performance decay (5 minutes)
- Approaches we tried: new hyperparameters, weighted anchor data, incremental learning, inverse propensity weighting (IPW).
- Deep dive on IPW (5 minutes)
- What is IPW?
- Pure IPW vs Trees.
- How large sample weights cause gradient variance explosions, forcing the model to memorize outlier noise.
- Solution: Asymmetric IPW (10 minutes)
- Asymmetry:
- Anchor True Negative sample weights to 1.0 (holdout and non-holdout data).
- Use Optuna to optimize a weighting parameter for False Negatives (holdout and non-holdout data).
- Use Optuna to optimize a downscaling multiplier for the True Positives and False Positives IPW weights (holdout exclusive data).
- Stabilization:
- Protect rare minority classes (TPs, FPs) from random erasure using stratified downsampling.
- Setup robust validation strategy and optimize regularization hyperparameters to control for overfitting/underfitting.
- Performance evaluation across all approaches
- Evaluation metric: Partial AUC @20% FPR.
- Solution recovers 99% of benchmark's performance (model trained on fully uncensored dataset).
- Yields a ~17% improvement in partial AUC compared to a baseline of no retraining (model decay).
- Yields a 5~7% improvement in partial AUC compared to the alternative retraining methodologies tested.
- Q&A (5 minutes)
Senior Data Scientist based in Tallinn, Estonia, with 9+ years of experience in machine learning, analytics, experimentation, and causal inference.