Real-world applications utilize machine learning (or artificial intelligence) to assist in decision-making or planning. In this context, probabilistic models represent the standard approach for data scientists to link input data with probabilistic predictions for possible outcomes, which serve as the foundation for subsequent operational decisions or actions. An additional challenge encountered in real-world applications is that the outcome of interest to be predicted often occurs rarely compared to the total number of observations. This scenario is commonly referred to as an "imbalanced" dataset. Several examples illustrate such applications: (i) medical screening where detecting a specific disease represents a rare outcome compared to the general population, (ii) fraud detection where fraudulent events constitute a small fraction of total transactions, and (iii) detection of physical phenomena such as the Higgs boson where observations confirming its existence represent rare events compared to all observations.

This tutorial addresses this particular problem and examines specific methodological considerations when approaching imbalanced datasets. One important consideration relates to resampling and its various effects: with numerous negative outcomes compared to positive outcomes, existing literature supports reducing the ratio between these event types. With these techniques in mind, we examine and study the following aspects: (i) the impact of imbalanced datasets on probability outcomes and classifier calibration; (ii) the effect of resampling on model overfitting and underfitting and its relationship to model regularization; and (iii) the impact of resampling on computational performance versus statistical performance.

This tutorial provides a hands-on approach through programmatic notebooks to offer insights into each of these concepts.