Principal Component Networks: Parameter Reduction Early in Training
Principal Component Networks: Parameter Reduction Early in Training
复制标题
DOI：
--
复制
发表时间：
2020-06
期刊：
影响因子：
--
通讯作者：
R. Waleffe;Theodoros Rekatsinas
中科院分区：
文献类型：
--
作者：
R. Waleffe;Theodoros Rekatsinas
最近的研究表明，过参数化网络包含较小的子网络，在隔离训练时，这些子网络表现出与完整模型相当的准确性。这些结果强调了在不牺牲泛化性能的情况下降低深度神经网络训练成本的潜力。然而，现有的寻找这些小网络的方法依赖于昂贵的多轮训练和修剪过程，并且对于大型数据集和模型是不实用的。在本文中，我们展示了如何找到小型网络，这些网络在经过几次训练后表现出与超参数化网络相同的性能。我们发现超参数化网络中的隐层激活主要存在于小于实际模型宽度的子空间中。在此基础上，我们使用PCA为层输入找到高方差的基础，并使用这些方向表示层权重。我们消除了所有与PCA基础无关的权重，并将这些网络架构称为主元网络。在CIFAR-10和ImageNet上，我们证明了PCN比过参数化模型训练得更快，使用的能量更少，而且没有准确性损失。我们发现，我们的转换导致网络的参数减少了23.8倍，具有相等或更高的最终模型精度-在某些情况下，我们观察到高达3%的改进。我们还表明，ResNet-20 PCN的性能优于深度ResNet-110网络，同时训练速度更快。
Recent works show that overparameterized networks contain small subnetworks that exhibit comparable accuracy to the full model when trained in isolation. These results highlight the potential to reduce training costs of deep neural networks without sacrificing generalization performance. However, existing approaches for finding these small networks rely on expensive multi-round train-and-prune procedures and are non-practical for large data sets and models. In this paper, we show how to find small networks that exhibit the same performance as their overparameterized counterparts after only a few training epochs. We find that hidden layer activations in overparameterized networks exist primarily in subspaces smaller than the actual model width. Building on this observation, we use PCA to find a basis of high variance for layer inputs and represent layer weights using these directions. We eliminate all weights not relevant to the found PCA basis and term these network architectures Principal Component Networks. On CIFAR-10 and ImageNet, we show that PCNs train faster and use less energy than overparameterized models, without accuracy loss. We find that our transformation leads to networks with up to 23.8x fewer parameters, with equal or higher end-model accuracy---in some cases we observe improvements up to 3%. We also show that ResNet-20 PCNs outperform deep ResNet-110 networks while training faster.