Dropout: Explicit Forms and Capacity Control
Dropout: Explicit Forms and Capacity Control
复制标题
DOI：
--
复制
发表时间：
2020-03
期刊：
影响因子：
--
通讯作者：
R. Arora;P. Bartlett;Poorya Mianjy;N. Srebro
中科院分区：
文献类型：
--
作者：
R. Arora;P. Bartlett;Poorya Mianjy;N. Srebro
我们研究了各种机器学习问题中由dropout提供的容量控制。首先，我们研究了矩阵完备化的dropout，在那里它诱导了一个依赖于数据的正则化子，在期望中，它等于因子乘积的加权迹范数。在深度学习中，我们证明了由于dropout导致的数据依赖正则化器直接控制了底层深度神经网络的Rademacher复杂度。这些发展使我们能够在矩阵补全和训练深度神经网络中为dropout算法给出具体的泛化误差界。我们在真实世界的数据集上评估了我们的理论研究结果，包括MovieLens，MNIST和Fashion-MNIST。
We investigate the capacity control provided by dropout in various machine learning problems. First, we study dropout for matrix completion, where it induces a data-dependent regularizer that, in expectation, equals the weighted trace-norm of the product of the factors. In deep learning, we show that the data-dependent regularizer due to dropout directly controls the Rademacher complexity of the underlying class of deep neural networks. These developments enable us to give concrete generalization error bounds for the dropout algorithm in both matrix completion as well as training deep neural networks. We evaluate our theoretical findings on real-world datasets, including MovieLens, MNIST, and Fashion-MNIST.