Provable Identifiability of Two-Layer ReLU Neural Networks via LASSO Regularization
Provable Identifiability of Two-Layer ReLU Neural Networks via LASSO Regularization
复制标题
DOI：
10.1109/tit.2023.3274152
复制
发表时间：
2023-05
影响因子：
2.5
通讯作者：
Geng Li;G. Wang;Jie Ding
中科院分区：
文献类型：
--
作者：
Geng Li;G. Wang;Jie Ding
LASSO正则化是一种流行的回归工具，它通过$\ well _{1}$惩罚来执行变量选择，从而提高统计模型的预测精度，该惩罚最初是为线性模型及其变体制定的。本文将LASSO的领域扩展到两层ReLU神经网络，这是一种新颖而强大的非线性回归模型。具体来说，给定一个神经网络，其输出$y$仅依赖于输入$\boldsymbol {x}$的一个小子集，表示为$\mathcal {S}^{\star}$，我们证明了LASSO估计器可以稳定地重建神经网络，并在样本数量随输入维数呈对数尺度变化时识别出$\mathcal {S}^{\star}$。这种具有挑战性的制度已经很好地理解了线性模型，而很少研究神经网络。我们的理论是一个扩展的基于限制等距特性（RIP）的双层ReLU神经网络分析框架，这可能对其他LASSO或神经网络设置有独立的兴趣。在此基础上，提出了一种基于神经网络的变量选择方法。在模拟和现实数据集上的实验表明，与现有的变量选择方法相比，该方法具有良好的性能。
LASSO regularization is a popular regression tool to enhance the prediction accuracy of statistical models by performing variable selection through the $\ell _{1}$ penalty, initially formulated for the linear model and its variants. In this paper, the territory of LASSO is extended to two-layer ReLU neural networks, a fashionable and powerful nonlinear regression model. Specifically, given a neural network whose output $y$ depends only on a small subset of input $\boldsymbol {x}$ , denoted by $\mathcal {S}^{\star }$ , we prove that the LASSO estimator can stably reconstruct the neural network and identify $\mathcal {S}^{\star }$ when the number of samples scales logarithmically with the input dimension. This challenging regime has been well understood for linear models while barely studied for neural networks. Our theory lies in an extended Restricted Isometry Property (RIP)-based analysis framework for two-layer ReLU neural networks, which may be of independent interest to other LASSO or neural network settings. Based on the result, we advocate a neural network-based variable selection method. Experiments on simulated and real-world datasets show promising performance of the variable selection approach compared with existing techniques.