Multicoated Supermasks Enhance Hidden Networks
Multicoated Supermasks Enhance Hidden Networks
复制标题
DOI：
--
复制
发表时间：
2022
期刊：
影响因子：
--
通讯作者：
Yasuyuki Okoshi;Ángel López García-Arias;Kazutoshi Hirose;Kota Ando;Kazushi Kawamura;Thiem Van Chu;M. Motomura;Jaehoon Yu
中科院分区：
文献类型：
--
作者：
Yasuyuki Okoshi;Ángel López García-Arias;Kazutoshi Hirose;Kota Ando;Kazushi Kawamura;Thiem Van Chu;M. Motomura;Jaehoon Yu
隐藏网络(Ramanujan等人，2020年)展示了通过训练连接掩码(称为超级掩码)在随机加权神经网络中找到精确子网络的可能性。我们发现，即使梯度不为零，超级掩码也会停止改进，从而未充分利用反向传播的信息。为了解决这个问题，我们提出了一种扩展隐藏网络的方法，通过训练多个分层超级掩码的叠加-多涂层超级掩码来扩展隐藏网络。该方法表明，在不增加训练代价的情况下，对单个任务使用多个超级掩码可以获得更高的准确率。在CIFAR-10和ImageNet上的实验表明，多涂层超级掩模增强了精度和模型尺寸之间的权衡。使用7层超级掩模的ResNet-101的性能比其对应的Hidden Networks高出4%，与密集ResNet-50的精度相当，同时要小一个数量级。
Hidden Networks (Ramanujan et al., 2020) showed the possibility of finding accurate sub-networks within a randomly weighted neural network by training a connectivity mask, referred to as supermask. We show that the supermask stops improving even though gradients are not zero, thus underutilizing backpropagated information. To address this issue, we propose a method that extends Hidden Networks by training an overlay of multiple hierarchical supermasks— a Multicoated Supermask. This method shows that using multiple supermasks for a single task achieves higher accuracy without additional training cost. Experiments on CIFAR-10 and ImageNet show that Multicoated Supermasks enhance the tradeoff between accuracy and model size. A ResNet-101 using a 7-coated supermask outperforms its Hidden Networks counterpart by 4% , matching the accuracy of a dense ResNet-50 while being an order of magnitude smaller.