TabICL vs scikit-learn for predictive uncertainty quantification
TabICL is a fully-open source, state of the art Tabular Foundational Model (TFM). TFMs are pretrained on large amount of synthetic data and have been advertised as being able to provide well calibrated probabilistic predictions both for classification and regression tasks.
In this presentation, we will present the results of a comparative study of the predictive capabilities of TabICL versus several scikit-learn based alternatives. In particular, we will contrast the following aspects: simplicity of development, computational and statistical performance in small and large sample size regimes. Statistical performance will be evaluated with strictly proper scoring rules that assess both the calibration and sharpness of predictions on left out data.
We will also report on attempts to disentangle epistemic from aleatoric predictive uncertainty, both in homoscedastic and heteroscedastic settings and explain why this can be important for industrial applications where the machine learning models are used as surrogate models in a Bayesian optimization loop.
Olivier Grisel is a machine learning engineer at Probabl and a contributor to the scikit-learn library.