AF: RI: Medium: Collaborative Research: Understanding and Improving Optimization in Deep and Recurrent Networks

使用深度神经网络的机器学习最近在经验上取得了广泛的成功。尽管取得了这一成功，但人们对深度神经网络与数据相适应的优化过程仍然知之甚少。除了在将深度神经网络拟合到数据中发挥关键作用外，优化还强烈影响模型从训练示例推广到未知数据的能力。该项目将为大型人工神经网络为什么以及何时训练和泛化良好建立一个工作理论，并利用该理论开发新的优化方法。新方法的实用性将在涉及语言、语音、生物序列和其他序列数据的应用中得到证明。该项目将包括对研究生和本科生的培训，项目负责人将为机器学习社区以及其他使用机器学习工具的研究人员和工程师提供教程。为了建立一个理论，说明为什么以及何时非凸优化在训练深度网络时效果良好，将采用经验自上而下和分析自下而上的方法。自上而下的方法将涉及对实践中使用的大规模深度模型的现象学分析，无论是在呈现真实数据时，还是在呈现专门用于测试网络行为的数据时。自下而上的方法将涉及从越来越复杂的模型中进行精确的分析研究，从线性模型和非凸矩阵分解开始，通过线性神经网络，具有少量隐藏层的模型，最终达到更深更复杂的网络。所开发的理论旨在既具有解释性又具有可操作性，并将用于派生新的优化方法和对架构的修改，以帮助优化和泛化。一个特别重要的试验台是递归神经网络。递归神经网络是一种功能强大的序列模型，它在处理输入序列时保持状态，并用于序列数据。尤其具有挑战性的是，循环神经网络仍然为更强的原则性理解留下了很大的空间，而这正是该项目旨在提供的。该奖项反映了美国国家科学基金会的法定使命，并通过使用基金会的知识价值和更广泛的影响审查标准进行评估，被认为值得支持。

Machine learning using deep neural networks has recently demonstrated broad empirical success. Despite this success, the optimization procedures that fit deep neural networks to data are still poorly understood. Besides playing a crucial role in fitting deep neural networks to data, optimization also strongly affects the model's ability to generalize from training examples to unseen data. This project will establish a working theory for why and when large artificial neural networks train and generalize well, and use this theory to develop new optimization methods. The utility of the new methods will be demonstrated in applications involving language, speech, biological sequences and other sequence data. The project will involve training of graduate and undergraduate students, and the project leaders will offer tutorials aimed at both the machine learning community, and other researchers and engineers using machine learning tools. In order to establish a theory of why and when non-convex optimization works well when training deep networks, both empirical top-down and analytic bottom-up approaches will be pursued. The top-down approach will involve phenomenological analysis of large scale deep models used in practice, both when presented with real data, and when presented with data specifically crafted to test the behavior of the network. The bottom-up approach will involve precise analytic investigation from increasingly more complex models, starting with linear models, and non-convex matrix factorization, progressing through linear neural networks, models with a small number of hidden layers, and eventually reaching deeper and more complex networks. The theory developed aims to be both explanatory and actionable, and will be used to derive new optimization methods and modifications to architectures that aid in optimization and generalization. A particularly important testbed is the case of recurrent neural networks. Recurrent neural networks are powerful sequence models that maintain state as they process an input sequence and are used for sequence data. Particularly challenging to optimize, recurrent neural networks still leave much room for a stronger principled understanding, which the project aims to provide.This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.