LocalDrop: A Hybrid Regularization for Deep Neural Networks
LocalDrop: A Hybrid Regularization for Deep Neural Networks
复制标题
DOI：
10.1109/tpami.2021.3061463
复制
发表时间：
2021-02
影响因子：
23.6
通讯作者：
Ziqing Lu;Chang Xu-;Bo Du;Takashi Ishida;L. Zhang;Masashi Sugiyama
中科院分区：
文献类型：
--
作者：
Ziqing Lu;Chang Xu-;Bo Du;Takashi Ishida;L. Zhang;Masashi Sugiyama
在神经网络中，发展正则化算法来解决过拟合问题是一个重要的研究领域。我们提出了一种新的神经网络正则化的局部Rademacher复杂性称为LocalDrop。基于提出的局部Rademacher复杂度上界，通过严格的数学推导，提出了一种新的正则化函数，包括丢弃率和权值矩阵，用于全连接网络（FCN）和卷积神经网络（CNN）.在复杂度分析中还包括了在不同层保持速率矩阵的情况下，对FCNs中的丢弃和CNN中的丢弃块的分析。利用新的正则化函数，我们建立了一个两阶段的过程来获得最优的保持率矩阵和权重矩阵，以实现整个训练模型。大量的实验已经进行了证明LocalDrop在不同的模型中的有效性，通过比较它与几种算法和不同的超参数对最终性能的影响。
In neural networks, developing regularization algorithms to settle overfitting is one of the major study areas. We propose a new approach for the regularization of neural networks by the local Rademacher complexity called LocalDrop. A new regularization function for both fully-connected networks (FCNs) and convolutional neural networks (CNNs), including drop rates and weight matrices, has been developed based on the proposed upper bound of the local Rademacher complexity by the strict mathematical deduction. The analyses of dropout in FCNs and DropBlock in CNNs with keep rate matrices in different layers are also included in the complexity analyses. With the new regularization function, we establish a two-stage procedure to obtain the optimal keep rate matrix and weight matrix to realize the whole training model. Extensive experiments have been conducted to demonstrate the effectiveness of LocalDrop in different models by comparing it with several algorithms and the effects of different hyperparameters on the final performances.