Path Regularization: A Convexity and Sparsity Inducing Regularization for Parallel ReLU Networks-猫眼课题宝

DOI：

发表时间：

2021-10

期刊：

ArXiv

影响因子：

通讯作者：

中科院分区：

其他

文献类型：

作者：

Tolga Ergen;Mert Pilanci

了解深神经网络成功背后的基本原理是当前文献中最重要的开放问题之一。为此，我们研究了深神经网络的训练问题，并引入了一种分析方法，以在优化景观中揭示隐藏的凸度。我们考虑了深层的Relu网络体系结构，其中还包括标准的深网和重新连接作为特殊情况。然后，我们证明可以将正则化训练问题表示为确切的凸优化问题。我们进一步证明了等效凸问题是通过诱导规范的组稀疏性正规的。因此，可以将路径正规的并行relu网络视为高尺寸的简约凸模型。更重要的是，由于原始训练问题可能无法在多项式时间内训练，因此我们提出了一种近似算法，在所有数据维度中具有完全多项式时间的复杂性。然后，我们证明了该算法可确保强大的全球最优性。我们还提供了证实理论的实验。

Understanding the fundamental principles behind the success of deep neural networks is one of the most important open questions in the current literature. To this end, we study the training problem of deep neural networks and introduce an analytic approach to unveil hidden convexity in the optimization landscape. We consider a deep parallel ReLU network architecture, which also includes standard deep networks and ResNets as its special cases. We then show that pathwise regularized training problems can be represented as an exact convex optimization problem. We further prove that the equivalent convex problem is regularized via a group sparsity inducing norm. Thus, a path regularized parallel ReLU network can be viewed as a parsimonious convex model in high dimensions. More importantly, since the original training problem may not be trainable in polynomial-time, we propose an approximate algorithm with a fully polynomial-time complexity in all data dimensions. Then, we prove strong global optimality guarantees for this algorithm. We also provide experiments corroborating our theory.

Path Regularization: A Convexity and Sparsity Inducing Regularization for Parallel ReLU Networks

文献摘要

相似文献