Stable Rank Normalization for Improved Generalization in Neural Networks and GANs
Stable Rank Normalization for Improved Generalization in Neural Networks and GANs
复制标题
DOI：
--
复制
发表时间：
2019-06
期刊：
影响因子：
--
通讯作者：
Amartya Sanyal;Philip H. S. Torr;P. Dokania
中科院分区：
文献类型：
--
作者：
Amartya Sanyal;Philip H. S. Torr;P. Dokania
Neyshabur等人对神经网络（NN）的推广界进行了令人兴奋的新工作，Bartlett等人密切依赖于两个参数依赖的量：Lipschitz常数上限和稳定秩（秩算子的较软版本）。这就引出了一个有趣的问题，即控制这些量是否可以改善NN的泛化行为。为此，我们提出了稳定秩归一化（SRN），一种新颖的，最佳的，计算效率高的权重归一化方案，最大限度地减少了线性算子的稳定秩。令人惊讶的是，我们发现，SRN，尽管是非凸问题，可以被证明有一个唯一的最优解。此外，我们表明，SRN允许控制数据依赖的经验Lipschitz常数，这与Lipschitz上限相反，反映了给定数据集上模型的真实行为。我们提供了彻底的分析表明，SRN，当应用到一个NN的线性层进行分类，提供了显着的改善-11.3%的泛化差距相比，标准NN沿着显着减少记忆。当应用于GAN的迭代（称为SRN-GAN）时，它可以提高CIFAR 10/100和CelebA数据集上的Inception，FID和Neural divergence分数，同时学习具有低经验Lipschitz常数的映射。
Exciting new work on the generalization bounds for neural networks (NN) given by Neyshabur et al. , Bartlett et al. closely depend on two parameter-depenedent quantities: the Lipschitz constant upper-bound and the stable rank (a softer version of the rank operator). This leads to an interesting question of whether controlling these quantities might improve the generalization behaviour of NNs. To this end, we propose stable rank normalization (SRN), a novel, optimal, and computationally efficient weight-normalization scheme which minimizes the stable rank of a linear operator. Surprisingly we find that SRN, inspite of being non-convex problem, can be shown to have a unique optimal solution. Moreover, we show that SRN allows control of the data-dependent empirical Lipschitz constant, which in contrast to the Lipschitz upper-bound, reflects the true behaviour of a model on a given dataset. We provide thorough analyses to show that SRN, when applied to the linear layers of a NN for classification, provides striking improvements-11.3% on the generalization gap compared to the standard NN along with significant reduction in memorization. When applied to the discriminator of GANs (called SRN-GAN) it improves Inception, FID, and Neural divergence scores on the CIFAR 10/100 and CelebA datasets, while learning mappings with low empirical Lipschitz constants.