Learning Compact Representations of Neural Networks using DiscriminAtive Masking (DAM)-猫眼课题宝

DOI：

发表时间：

2021-10

期刊：

影响因子：

通讯作者：

中科院分区：

其他

文献类型：

作者：

Jie Bu;Arka Daw;M. Maruf;A. Karpatne

深度学习的一个核心目标是学习神经网络每一层特征的紧凑表示，这对于无监督表示学习和结构化网络修剪都很有用。虽然在结构化修剪方面的工作越来越多，但当前最先进的方法受到两个关键限制：（i）训练期间的不稳定性，以及（ii）需要额外的微调步骤，这是资源密集型的。这些限制的核心是缺乏一种系统的方法，在单个阶段的训练过程中联合修剪和细化权重，并且不需要在收敛时进行任何微调以实现最先进的性能。我们提出了一种新的单阶段结构化剪枝方法，称为歧视性掩蔽（DAM）。DAM背后的关键直觉是在训练过程中有区别地选择一些神经元进行优化，同时逐渐掩盖其他神经元。我们表明，我们提出的DAM方法在各种应用中具有非常好的性能，包括降维，推荐系统，图表示学习和结构化修剪图像分类。我们还从理论上证明了DAM的学习目标与最小化掩蔽层的L0范数直接相关。

A central goal in deep learning is to learn compact representations of features at every layer of a neural network, which is useful for both unsupervised representation learning and structured network pruning. While there is a growing body of work in structured pruning, current state-of-the-art methods suffer from two key limitations: (i) instability during training, and (ii) need for an additional step of fine-tuning, which is resource-intensive. At the core of these limitations is the lack of a systematic approach that jointly prunes and refines weights during training in a single stage, and does not require any fine-tuning upon convergence to achieve state-of-the-art performance. We present a novel single-stage structured pruning method termed DiscriminAtive Masking (DAM). The key intuition behind DAM is to discriminatively prefer some of the neurons to be refined during the training process, while gradually masking out other neurons. We show that our proposed DAM approach has remarkably good performance over various applications, including dimensionality reduction, recommendation system, graph representation learning, and structured pruning for image classification. We also theoretically show that the learning objective of DAM is directly related to minimizing the L0 norm of the masking layer.

Learning Compact Representations of Neural Networks using DiscriminAtive Masking (DAM)

文献摘要

相似文献