EAGER: Quantifying the error landscape of deep neural networks

深度学习系统在广泛的应用中取得的巨大成功可以归因于它们能够很好地近似复杂函数，它们能够有效地进行训练，以及它们在预测未知输入值方面的良好表现。最后一个属性，被称为泛化，特别令人困惑。据观察，由称为随机梯度下降的优化算法训练的深度神经网络（DNN）产生的模型泛化能力很好，特别是当模型参数的数量大大超过模型训练的样本数量时。传统理论无法解释这些现象，需要新的视角和研究手段来阐明这些现象。为此，统计力学可以提供能够解决深度学习中长期存在的问题的方法和观点。能量景观代表了这些领域交叉点的一种常见范式：当训练DNN时，我们将所谓的“误差景观”降到与模型参数的特定选择相对应的最小值。理解DNN中的泛化性能相当于理解错误景观的结构和训练算法的动态之间的相互作用。特别是，“平坦最小值”的概念越来越流行，作为这些观察结果的可能解释，但缺乏估计平坦度的严格方法。我们建议采用统计力学中开发的一类新方法来回答有关DNN错误景观结构的问题，并确定找到给定解决方案的概率，其平坦性及其泛化性能之间的关系。这条研究路线应该对我们理解深度学习系统中的泛化产生重大影响，并对运输，安全和医学等高风险应用产生影响。该提案旨在为DNN的错误景观的表征以及景观结构和优化动态之间的相互作用如何产生可推广的解决方案带来新的严格程度。因此，我们将能够阐明为什么DNN具有低估计误差（即，高泛化性能）。这样的理解将代表着深度学习理论发展的重要一步。我们的目标是这样做，利用国家的科学数值技术来衡量的吸引力在高维参数空间盆地的体积。我们将测量流域体积分布和相关的平坦度，作为网络参数数量和泛化性能的函数。该奖项反映了NSF的法定使命，并通过使用基金会的知识价值和更广泛的影响审查标准进行评估，被认为值得支持。

The remarkable success achieved by deep learning systems in a broad number of applications can be attributed to their ability to approximate complex functions well, their aptitude to being trained efficiently, and their good performance in predicting the values of unseen inputs. This last property, known as generalization, is particularly puzzling. It is observed that deep neural networks (DNNs) trained by the optimization algorithm known as stochastic gradient descent produce models that generalize well, particularly when the number of model parameters greatly exceeds the number of samples on which the model is trained. Traditional theory fails to explain these observations and new perspectives and means of investigation are necessary to elucidate these phenomena. To this end, statistical mechanics may provide methods and perspectives capable of addressing long-standing questions in deep learning. The energy landscape represents a common paradigm at the intersection of these fields: when training a DNN we descend the so- called “error landscape” towards a minimum corresponding to a particular choice of model parameters. Understanding generalization performance in DNNs amounts to understanding the interplay between the structure of the error landscape and the dynamics of the training algorithm that descends it. In particular, the concept of “flat minima” is gaining popularity as a possible explanation for these observations, but a rigorous approach for estimating flatness is lacking. We propose to employ a new class of methods developed within statistical mechanics to answer questions concerning the structure of the error landscapes of DNNs and to identify the relationship between the probability of finding a given solution, its flatness and its generalization performance. This line of investigation should have a significant impact on our understanding of generalization in deep learning systems with implications for high-stakes applications such as transportation, security and medicine.This proposal seeks to bring a new degree of rigor in the characterization of the error landscape of DNNs and how the interplay between landscape structure and optimization dynamics yield generalizable solutions. As a result, we will be able to elucidate why DNNs are endowed with low estimation error (i.e., high generalization performance). Such an understanding will represent a significant step forward in the development of a theory of deep learning. We aim to do so by exploiting state-of-the-science numerical techniques to measure the volume of basins of attraction in high-dimensional parameter spaces. We will measure the basin volume distributions and the associated flatness as a function of the number of parameters and the generalization performance of the network.This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.