Complexity of Linear Regions in Deep Networks
Complexity of Linear Regions in Deep Networks
复制标题
DOI：
--
复制
发表时间：
2019-01
期刊：
影响因子：
--
通讯作者：
B. Hanin;D. Rolnick
中科院分区：
文献类型：
--
作者：
B. Hanin;D. Rolnick
众所周知，神经网络的表达能力取决于其架构，更深的网络表达更复杂的功能。在计算分段线性函数的网络中，例如具有ReLU激活的网络，不同线性区域的数量是表达能力的自然度量。我们可以只使用一个区域来构建网络，或者线性区域的数量随着深度呈指数增长;我们不清楚大多数网络在训练之前或之后都落在这个范围内。在本文中，我们提供了一个数学框架来计算分段线性网络的线性区域的数量，并测量这些区域之间的边界的体积。特别是，我们证明了网络在初始化时，区域的平均数量沿着任何一维子空间线性增长的神经元总数，远低于指数上限。我们还发现，在初始化尺度上，到最近区域边界的平均距离与神经元数量成反比。我们的理论表明，即使经过训练，线性区域的数量也远低于指数，这与我们的经验观察相匹配。我们的结论是，神经网络的实际表现力可能远远低于理论最大值，并且这种差距可以量化。
It is well-known that the expressivity of a neural network depends on its architecture, with deeper networks expressing more complex functions. In the case of networks that compute piecewise linear functions, such as those with ReLU activation, the number of distinct linear regions is a natural measure of expressivity. It is possible to construct networks with merely a single region, or for which the number of linear regions grows exponentially with depth; it is not clear where within this range most networks fall in practice, either before or after training. In this paper, we provide a mathematical framework to count the number of linear regions of a piecewise linear network and measure the volume of the boundaries between these regions. In particular, we prove that for networks at initialization, the average number of regions along any one-dimensional subspace grows linearly in the total number of neurons, far below the exponential upper bound. We also find that the average distance to the nearest region boundary at initialization scales like the inverse of the number of neurons. Our theory suggests that, even after training, the number of linear regions is far below exponential, an intuition that matches our empirical observations. We conclude that the practical expressivity of neural networks is likely far below that of the theoretical maximum, and that this gap can be quantified.