On the Implicit Bias of Initialization Shape: Beyond Infinitesimal Mirror Descent
On the Implicit Bias of Initialization Shape: Beyond Infinitesimal Mirror Descent
复制标题
DOI：
--
复制
发表时间：
2021-02
期刊：
影响因子：
--
通讯作者：
Shahar Azulay;E. Moroshko;M. S. Nacson;Blake E. Woodworth;N. Srebro;A. Globerson;Daniel Soudry
中科院分区：
文献类型：
--
作者：
Shahar Azulay;E. Moroshko;M. S. Nacson;Blake E. Woodworth;N. Srebro;A. Globerson;Daniel Soudry
最近的工作强调了初始化规模在确定梯度方法收敛的解决方案的结构中的作用。特别是，它表明，大的初始化导致的神经切线核政权的解决方案，而小的初始化导致所谓的“丰富的制度”。然而，初始化结构比单独的整体规模更丰富，并且涉及网络中不同权重和层的相对大小。在这里，我们表明，这些相对尺度，我们称之为初始化形状，在确定学习模型中起着重要的作用。我们开发了一种新的技术，用于推导梯度流的诱导偏差，并使用它来获得封闭形式的隐式正则化多个感兴趣的情况。
Recent work has highlighted the role of initialization scale in determining the structure of the solutions that gradient methods converge to. In particular, it was shown that large initialization leads to the neural tangent kernel regime solution, whereas small initialization leads to so called"rich regimes". However, the initialization structure is richer than the overall scale alone and involves relative magnitudes of different weights and layers in the network. Here we show that these relative scales, which we refer to as initialization shape, play an important role in determining the learned model. We develop a novel technique for deriving the inductive bias of gradient-flow and use it to obtain closed-form implicit regularizers for multiple cases of interest.