Taxonomizing local versus global structure in neural network loss landscapes
Taxonomizing local versus global structure in neural network loss landscapes
复制标题
DOI：
--
复制
发表时间：
2021-07
期刊：
影响因子：
--
通讯作者：
Yaoqing Yang;Liam Hodgkinson;Ryan Theisen;Joe Zou;Joseph Gonzalez;K. Ramchandran;Michael W. Mahoney-Michael-W.-Mah
中科院分区：
文献类型：
--
作者：
Yaoqing Yang;Liam Hodgkinson;Ryan Theisen;Joe Zou;Joseph Gonzalez;K. Ramchandran;Michael W. Mahoney-Michael-W.-Mah
从损失景观的角度来看待神经网络模型在统计力学学习方法中有着悠久的历史，近年来它在机器学习中受到了关注。除此之外，局部度量（例如损失景观的平滑度）已被证明与模型的全局属性（例如良好的泛化性能）相关。在这里，我们对数千个神经网络模型的损失景观结构进行了详细的实证分析，系统地改变了学习任务，模型架构和/或数据的数量/质量。通过考虑一系列试图捕捉损失景观不同方面的指标，我们证明了最佳的测试精度是在以下情况下获得的：损失景观是全局良好连接的;训练模型的集合彼此更相似;模型收敛到局部平滑区域。我们还表明，当模型很小或者当它们被训练成较低质量的数据时，可能会出现全局连接不良的景观;并且，如果损失景观是全局连接不良的，那么训练到零损失实际上会导致更差的测试准确性。我们详细的实证结果揭示了学习阶段（以及随之而来的双下降行为），良好泛化的基本与偶然决定因素，负载和温度参数在学习过程中的作用，模型和数据对损失景观的不同影响，以及局部和全局度量之间的关系，最近感兴趣的所有主题。
Viewing neural network models in terms of their loss landscapes has a long history in the statistical mechanics approach to learning, and in recent years it has received attention within machine learning proper. Among other things, local metrics (such as the smoothness of the loss landscape) have been shown to correlate with global properties of the model (such as good generalization performance). Here, we perform a detailed empirical analysis of the loss landscape structure of thousands of neural network models, systematically varying learning tasks, model architectures, and/or quantity/quality of data. By considering a range of metrics that attempt to capture different aspects of the loss landscape, we demonstrate that the best test accuracy is obtained when: the loss landscape is globally well-connected; ensembles of trained models are more similar to each other; and models converge to locally smooth regions. We also show that globally poorly-connected landscapes can arise when models are small or when they are trained to lower quality data; and that, if the loss landscape is globally poorly-connected, then training to zero loss can actually lead to worse test accuracy. Our detailed empirical results shed light on phases of learning (and consequent double descent behavior), fundamental versus incidental determinants of good generalization, the role of load-like and temperature-like parameters in the learning process, different influences on the loss landscape from model and data, and the relationships between local and global metrics, all topics of recent interest.