Regularizing Neural Networks via Minimizing Hyperspherical Energy
Regularizing Neural Networks via Minimizing Hyperspherical Energy
复制标题
DOI：
10.1109/cvpr42600.2020.00695
复制
发表时间：
2019-06
期刊：
影响因子：
--
通讯作者：
Rongmei Lin;Weiyang Liu;Zhen Liu;Chen Feng-;Zhiding Yu;J. Rehg;Li Xiong;Le Song
中科院分区：
文献类型：
--
作者：
Rongmei Lin;Weiyang Liu;Zhen Liu;Chen Feng-;Zhiding Yu;J. Rehg;Li Xiong;Le Song
受物理学中的Thomson问题的启发，超球面能量最小化问题可以通过最小化势能来模拟单位球面上多个推进电子的分布，它在正则化神经网络和提高其泛化能力方面显示出了潜在的潜力。本文首先通过对超球能量训练动力学的分析，研究了超球能量在神经网络训练中的重要作用。然后，我们证明了当空间维度变得更高时，由于高度的非线性和非凸最优化，单纯最小化超球面能量的方法遇到了一些困难，从而限制了进一步提高泛化的潜力。为了解决这些问题，我们提出了压缩最小超球面能量(CoMHE)作为一种更有效的神经网络正则化方法。具体地说，CoMHE利用投影映射来降低神经元的维度，并最小化其超球面能量。根据投影映射的不同设计，我们提出了几种截然不同但性能良好的变体，并为它们的有效性提供了一些理论保证。实验表明，CoMHE的性能始终优于现有的正则化方法，并且可以很容易地应用于不同的神经网络。
Inspired by the Thomson problem in physics where the distribution of multiple propelling electrons on a unit sphere can be modeled via minimizing some potential energy, hyperspherical energy minimization has demonstrated its potential in regularizing neural networks and improving their generalization power. In this paper, we first study the important role that hyperspherical energy plays in neural network training by analyzing its training dynamics. Then we show that naively minimizing hyperspherical energy suffers from some difficulties due to highly non-linear and non-convex optimization as the space dimensionality becomes higher, therefore limiting the potential to further improve the generalization. To address these problems, we propose the compressive minimum hyperspherical energy (CoMHE) as a more effective regularization for neural networks. Specifically, CoMHE utilizes projection mappings to reduce the dimensionality of neurons and minimizes their hyperspherical energy. According to different designs for the projection mapping, we propose several distinct yet well-performing variants and provide some theoretical guarantees to justify their effectiveness. Our experiments show that CoMHE consistently outperforms existing regularization methods, and can be easily applied to different neural networks.