Depthwise Hyperparameter Transfer in Residual Networks: Dynamics and Scaling Limit
Depthwise Hyperparameter Transfer in Residual Networks: Dynamics and Scaling Limit
复制标题
DOI：
10.48550/arxiv.2309.16620
复制
发表时间：
2023-09
期刊：
影响因子：
--
通讯作者：
Blake Bordelon;Lorenzo Noci;Mufan Bill Li;Boris Hanin;C. Pehlevan
中科院分区：
文献类型：
--
作者：
Blake Bordelon;Lorenzo Noci;Mufan Bill Li;Boris Hanin;C. Pehlevan
深度学习中超参数调优的成本随着模型规模的增加而上升，这促使从业者使用较小网络的代理来寻找新的调优方法。其中一个建议使用$\mu$ P参数化网络，其中小宽度网络的最优超参数转移到任意大宽度的网络。然而，在这个方案中，超参数不能跨深度传输。作为补救措施，我们结合$\mu$ P参数化研究残差分支尺度为$1/\sqrt{\text{depth}}$的残差网络。我们提供的实验表明，使用该参数化训练的残差架构（包括卷积ResNets和Vision transformer）在CIFAR-10和ImageNet上表现出跨宽度和深度的最优超参数转移。此外，我们的实证研究结果得到了理论的支持和激励。利用动态平均场理论（DMFT）描述神经网络学习动力学的最新发展，我们表明ResNets的这种参数化承认一个定义良好的特征学习联合无限宽和无限深极限，并显示有限大小的网络动力学向该极限收敛。
The cost of hyperparameter tuning in deep learning has been rising with model sizes, prompting practitioners to find new tuning methods using a proxy of smaller networks. One such proposal uses $\mu$P parameterized networks, where the optimal hyperparameters for small width networks transfer to networks with arbitrarily large width. However, in this scheme, hyperparameters do not transfer across depths. As a remedy, we study residual networks with a residual branch scale of $1/\sqrt{\text{depth}}$ in combination with the $\mu$P parameterization. We provide experiments demonstrating that residual architectures including convolutional ResNets and Vision Transformers trained with this parameterization exhibit transfer of optimal hyperparameters across width and depth on CIFAR-10 and ImageNet. Furthermore, our empirical findings are supported and motivated by theory. Using recent developments in the dynamical mean field theory (DMFT) description of neural network learning dynamics, we show that this parameterization of ResNets admits a well-defined feature learning joint infinite-width and infinite-depth limit and show convergence of finite-size network dynamics towards this limit.