Improved generalization bounds of group invariant / equivariant deep networks via quotient feature spaces
Improved generalization bounds of group invariant / equivariant deep networks via quotient feature spaces
复制标题
DOI：
--
复制
发表时间：
2019-10
期刊：
影响因子：
--
通讯作者：
Akiyoshi Sannai;M. Imaizumi;M. Kawano
中科院分区：
文献类型：
--
作者：
Akiyoshi Sannai;M. Imaizumi;M. Kawano
许多不变(或等变)神经网络已经成功地处理了不变数据，如点云和图形。然而，神经网络的泛化理论还没有得到很好的发展，因为该理论的几个基本因素，如网络规模和边际分布，与不变性和等方差没有深入的联系。在这项研究中，我们提出了一种新的不变和等变深度神经网络的泛化误差界。为了描述不变性和等方差对泛化的影响，我们提出了一个文本商特征空间的概念，它度量了群操作对属性的影响。我们的主要结果证明了商特征空间的体积可以描述泛化误差。此外，界的不变性和等方差显著地改善了界的前导项。我们将我们的结果应用于特定的不变和等变网络，如DeepSets(Zaheer等人。(2017))，并证明了它们的广义界有很大的改善，其中$n！$是排列的个数。我们还讨论了不变DNN的表达能力，并证明了它们可以达到最优的逼近速度。我们的实验结果支持我们的理论主张。
Numerous invariant (or equivariant) neural networks have succeeded in handling invariant data such as point clouds and graphs. However, a generalization theory for the neural networks has not been well developed, because several essential factors for the theory, such as network size and margin distribution, are not deeply connected to the invariance and equivariance. In this study, we develop a novel generalization error bound for invariant and equivariant deep neural networks. To describe the effect of invariance and equivariance on generalization, we develop a notion of a \textit{quotient feature space}, which measures the effect of group actions for the properties. Our main result proves that the volume of quotient feature spaces can describe the generalization error. Furthermore, the bound shows that the invariance and equivariance significantly improve the leading term of the bound. We apply our result to specific invariant and equivariant networks, such as DeepSets (Zaheer et al. (2017)), and show that their generalization bound is considerably improved by $\sqrt{n!}$, where $n!$ is the number of permutations. We also discuss the expressive power of invariant DNNs and show that they can achieve an optimal approximation rate. Our experimental result supports our theoretical claims.