On the Sensitivity of the Lasso to the Number of Predictor Variables
On the Sensitivity of the Lasso to the Number of Predictor Variables
复制标题
DOI：
10.1214/16-sts586
复制
发表时间：
2017-02-01
影响因子：
5.7
通讯作者：
Simonoff, Jeffrey S.
中科院分区：
文献类型：
--
作者：
Flynn, Cheryl J.;Hurvich, Clifford M.;Simonoff, Jeffrey S.
Lasso是一种计算效率很高的回归正则化过程，当预测量(p)很大时，它可以产生稀疏估计量。Oracle不等式在正则化参数的确定性选择下为Lasso估计器提供了概率损失界。如果p得到适当的控制，这些边界趋于零，因此通常被引用为Lasso及其处理高维设置的能力的理论依据。不幸的是，在实践中，正则化参数没有被选择为确定性的数量，而是使用随机的、依赖于数据的过程来选择。为了解决先前理论工作的这一缺点，我们研究了Lasso估计器在优化预测时的损失。假设正交预测器和稀疏真模型，我们证明了Lasso的最佳预测性能随着p的增加而恶化的概率是正的，并且可以任意接近一个给定足够高的信噪比和足够大的p。我们进一步证明了性能恶化的数量可能比oracle不等式所建议的要差得多，并提供了一个观察到恶化的真实数据示例。
The Lasso is a computationally efficient regression regularization procedure that can produce sparse estimators when the number of predictors (p) is large. Oracle inequalities provide probability loss bounds for the Lasso estimator at a deterministic choice of the regularization parameter. These bounds tend to zero if p is appropriately controlled, and are thus commonly cited as theoretical justification for the Lasso and its ability to handle high-dimensional settings. Unfortunately, in practice the regularization parameter is not selected to be a deterministic quantity, but is instead chosen using a random, data-dependent procedure. To address this shortcoming of previous theoretical work, we study the loss of the Lasso estimator when tuned optimally for prediction. Assuming orthonormal predictors and a sparse true model, we prove that the probability that the best possible predictive performance of the Lasso deteriorates as p increases is positive and can be arbitrarily close to one given a sufficiently high signal to noise ratio and sufficiently large p. We further demonstrate empirically that the amount of deterioration in performance can be far worse than the oracle inequalities suggest and provide a real data example where deterioration is observed.