Applied scientific research increasingly uses Fat-Data (e.g. large number of explanatory variables relative to number of observations) for feature selection purposes. Previous version of our all-subset-regression Julia package was unable to deal with such databases. Existing ML packages (e.g. Lasso.jl) overcome this problem paying a cost in terms of statistical inference, coefficient robustness and feature selection optimality (because ML algorithms focus on prediction not on explanation or causal-prediction). The new GlobalSearchRegression.jl version combines regularization pre-processsing with all-subset-regression algorithms to efficiently work with Fat Data without losing EC-strengths in terms of sensitivity analysis, residual properties and coefficient robustness.
In the first 3 minutes, our Lighting talk will discuss GlobalSearchRegression.jl new capabilities. We will focus on the main advantages of merging ML and EC algorithms for feature selection when the number of potential covariates is relatively large: ML provides efficiency and sub-sample uncertainty assessment while EC guarantees in-sample and out-of-sample optimality with covariate uncertainty assessment.
Then, we will show different benchmarks for the new GlobalSearchRegression.jl package against R and Stata counterparts, as well as against their own original version. Our updated ML-EC- algorithm written in Julia is up to 100 times faster than similar R or Stata programs, and allows working with hundreds of potential covariates (while the upper limit for the original GlobalSearchRegression.jl version was 28).
Finally, we will use the last 4 minutes for a live hands-on example to show the Graphical User Interface, execute the ML-EC algorithm with fat data and analyze main results using new output capabilities in Latex-PDF.