Approaches to Regularized Regression - A Comparison between Gradient Boosting and the Lasso
Approaches to Regularized Regression - A Comparison between Gradient Boosting and the Lasso
复制标题
DOI：
10.3414/me16-01-0033
复制
发表时间：
2016-01-01
影响因子：
1.7
通讯作者：
Mayr, Andreas
中科院分区：
文献类型：
--
作者：
Hepp, Tobias;Schmid, Matthias;Mayr, Andreas
背景：统计建模的惩罚和正则化技术由于其在高维数据中的优势，在生物医学研究中引起了越来越多的关注。特别关注的是包含自动变量选择的算法，如最小绝对收缩算子（lasso）或统计增强技术。目标：关注线性回归框架，本文从方法论和实践的角度比较了这一任务的两种最常用的技术，套索和梯度增强。方法：我们描述了这些方法，强调在哪些情况下，他们的结果将在低维设置一致。此外，我们进行了广泛的模拟研究，比较了在预测因子多于观测值的情况下的性能，并研究了信噪比和真非零系数数量的多种组合。最后，我们研究了不同调优方法对结果的影响。结果：两种方法都对可能的高维数据进行惩罚和变量选择，往往导致非常相似的模型。套索的一个优点是它更快的运行时间，增强概念的一个优点是它的模块化性质，使它很容易扩展到其他回归设置。结论：尽管在优化和正则化方面采用了不同的策略，但两种方法对估计问题的约束相似，因此在实践中，在预测精度和变量选择方面的性能相当。
Background: Penalization and regularization techniques for statistical modeling have attracted increasing attention in biomedical research due to their advantages in the presence of high-dimensional data. A special focus lies on algorithms that incorporate automatic variable selection like the least absolute shrinkage operator (lasso) or statistical boosting techniques.Objectives: Focusing on the linear regression framework, this article compares the two most-common techniques for this task, the lasso and gradient boosting, both from a methodological and a practical perspective.Methods: We describe these methods highlighting under which circumstances their results will coincide in low-dimensional settings. In addition, we carry out extensive simulation studies comparing the performance in settings with more predictors than observations and investigate multiple combinations of noise-to-signal ratio and number of true non-zero coeffcients. Finally, we examine the impact of different tuning methods on the results.Results: Both methods carry out penalization and variable selection for possibly highdimensional data, often resulting in very similar models. An advantage of the lasso is its faster run-time, a strength of the boosting concept is its modular nature, making it easy to extend to other regression settings.Conclusions: Although following different strategies with respect to optimization and regularization, both methods imply similar constraints to the estimation problem leading to a comparable performance regarding prediction accuracy and variable selection in practice.