Better Depth-Width Trade-offs for Neural Networks through the lens of Dynamical Systems
Better Depth-Width Trade-offs for Neural Networks through the lens of Dynamical Systems
复制标题
通过动态系统的视角更好地权衡神经网络的深度与宽度
DOI：
--
复制
发表时间：
2020
期刊：
影响因子：
--
通讯作者：
Ioannis Panageas
中科院分区：
文献类型：
--
作者：
Vaggos Chatziafratis;Sai Ganesh Nagarajan;Ioannis Panageas
神经网络的表达能力作为其深度、宽度和激活单元类型的函数一直是深度学习理论中的一个重要问题。最近，利用连续映射$f$的不动点的广义概念（称为周期点），通过与动力系统的新连接，获得了ReLU网络的深度分离结果。在这项工作中，我们加强了与动力系统的联系，并从几个方面改进了现有的宽度下界。我们的第一个主要结果是周期特定宽度下界，它适用于更强的$L^1$近似误差的概念，而不是更弱的分类误差。我们的第二个贡献是我们提供了更清晰的宽度下界，仍然产生有意义的指数深度-宽度分离，在以前的结果不适用的制度。我们的结果的一个副产品是，只要$f$具有奇数周期，就存在一个表征深度-宽度权衡的通用常数。从技术上讲，我们的结果揭示了一个给定函数的以下三个量之间的紧密联系：它的周期，它的Lipschitz常数和函数f与自身组合下产生的振荡次数的增长率。
The expressivity of neural networks as a function of their depth, width and type of activation units has been an important question in deep learning theory. Recently, depth separation results for ReLU networks were obtained via a new connection with dynamical systems, using a generalized notion of fixed points of a continuous map $f$, called periodic points. In this work, we strengthen the connection with dynamical systems and we improve the existing width lower bounds along several aspects. Our first main result is period-specific width lower bounds that hold under the stronger notion of $L^1$-approximation error, instead of the weaker classification error. Our second contribution is that we provide sharper width lower bounds, still yielding meaningful exponential depth-width separations, in regimes where previous results wouldn't apply. A byproduct of our results is that there exists a universal constant characterizing the depth-width trade-offs, as long as $f$ has odd periods. Technically, our results follow by unveiling a tighter connection between the following three quantities of a given function: its period, its Lipschitz constant and the growth rate of the number of oscillations arising under compositions of the function $f$ with itself.
深度 ReLU 网络的激活模式少得惊人
DOI：
--
发表时间：
2019
期刊：
Advances in neural information processing systems
影响因子：
--
作者：
Hanin, Boris;Rolnick, David
通讯作者：
Rolnick, David