Best Subset, Forward Stepwise or Lasso? Analysis and Recommendations Based on Extensive Comparisons
Best Subset, Forward Stepwise or Lasso? Analysis and Recommendations Based on Extensive Comparisons
复制标题
DOI：
10.1214/19-sts733
复制
发表时间：
2020-11-01
影响因子：
5.7
通讯作者：
Tibshirani, Ryan
中科院分区：
文献类型：
--
作者：
Hastie, Trevor;Tibshirani, Robert;Tibshirani, Ryan
在令人兴奋的最近的工作，Bertsimas，国王和Mazumder（安。44（2016）813-852）示出了回归建模中的经典最佳子集选择问题可以被公式化为混合整数优化（MIO）问题。利用MIO算法的最新进展，他们证明了最佳子集选择现在可以在比统计界认为可能的更大的问题规模上得到解决。他们提出了最佳子集与其他流行的变量选择程序，特别是套索和向前逐步选择的实证比较。令人惊讶的是，他们的模拟表明，最佳子集在预测准确性方面始终优于两种方法。在这里，我们提出了一组扩展的模拟，以更好地了解这些比较。总结大致如下：最佳子集和套索都不均匀地支配另一个，最佳子集通常在非常高的信噪比（SNR）状态下表现得更好，套索在低SNR状态下表现得更好;对于所考虑的大部分设置，最佳子集和前向逐步执行类似，但在高SNR状态下的某些情况下，最佳子集执行得更好;向前逐步和最佳子集往往产生稀疏模型（当在验证集上调谐时），特别是在高SNR状态中;松弛套索（实际上，是Meinshausen（Comput.中央集权主义者数据分析52（2007）374-393））是总的赢家，在低SNR场景中表现得几乎与套索一样好，并且在高SNR场景中几乎是最佳子集。
In exciting recent work, Bertsimas, King and Mazumder (Ann. Statist. 44 (2016) 813-852) showed that the classical best subset selection problem in regression modeling can be formulated as a mixed integer optimization (MIO) problem. Using recent advances in MIO algorithms, they demonstrated that best subset selection can now be solved at much larger problem sizes than what was thought possible in the statistics community. They presented empirical comparisons of best subset with other popular variable selection procedures, in particular, the lasso and forward stepwise selection. Surprisingly (to us), their simulations suggested that best subset consistently outperformed both methods in terms of prediction accuracy. Here, we present an expanded set of simulations to shed more light on these comparisons. The summary is roughly as follows:neither best subset nor the lasso uniformly dominate the other, with best subset generally performing better in very high signal-to-noise (SNR) ratio regimes, and the lasso better in low SNR regimes;for a large proportion of the settings considered, best subset and forward stepwise perform similarly, but in certain cases in the high SNR regime, best subset performs better;forward stepwise and best subsets tend to yield sparser models (when tuned on a validation set), especially in the high SNR regime;the relaxed lasso (actually, a simplified version of the original relaxed estimator defined in Meinshausen (Comput. Statist. Data Anal. 52 (2007) 374-393)) is the overall winner, performing just about as well as the lasso in low SNR scenarios, and nearly as well as best subset in high SNR scenarios.