AF: RI: Medium: Collaborative Research: Understanding and Improving Optimization in Deep and Recurrent Networks

AF：RI：中：协作研究：理解和改进深度和循环网络的优化

• 批准号: 1763562
• 负责人: Ruslan Salakhutdinov
• 金额: $ 32.91万
• 依托单位: Carnegie-Mellon University
• 依托单位国家: 美国
• 项目类别: Standard Grant
• 财政年份: 2018
• 资助国家: 美国
• 起止时间: 2018-08-01 至 2022-07-31
• 项目状态: 已结题
• 来源: https://www.nsf.gov/awardsearch/showAward?AWD_ID=1763562&HistoricalAwards=false
• 关键词: AF; RI; Medium; Collaborative; Research

Machine learning using deep neural networks has recently demonstrated broad empirical success. Despite this success, the optimization procedures that fit deep neural networks to data are still poorly understood. Besides playing a crucial role in fitting deep neural networks to data, optimization also strongly affects the model's ability to generalize from training examples to unseen data. This project will establish a working theory for why and when large artificial neural networks train and generalize well, and use this theory to develop new optimization methods. The utility of the new methods will be demonstrated in applications involving language, speech, biological sequences and other sequence data. The project will involve training of graduate and undergraduate students, and the project leaders will offer tutorials aimed at both the machine learning community, and other researchers and engineers using machine learning tools. In order to establish a theory of why and when non-convex optimization works well when training deep networks, both empirical top-down and analytic bottom-up approaches will be pursued. The top-down approach will involve phenomenological analysis of large scale deep models used in practice, both when presented with real data, and when presented with data specifically crafted to test the behavior of the network. The bottom-up approach will involve precise analytic investigation from increasingly more complex models, starting with linear models, and non-convex matrix factorization, progressing through linear neural networks, models with a small number of hidden layers, and eventually reaching deeper and more complex networks. The theory developed aims to be both explanatory and actionable, and will be used to derive new optimization methods and modifications to architectures that aid in optimization and generalization. A particularly important testbed is the case of recurrent neural networks. Recurrent neural networks are powerful sequence models that maintain state as they process an input sequence and are used for sequence data. Particularly challenging to optimize, recurrent neural networks still leave much room for a stronger principled understanding, which the project aims to provide.This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.

使用深度神经网络的机器学习最近已经证明了广泛的经验成功。尽管取得了这一成功，但将深度神经网络与数据相匹配的优化过程仍然知之甚少。除了在将深度神经网络拟合到数据中发挥关键作用外，优化还强烈影响模型从训练示例推广到未知数据的能力。该项目将建立一个工作理论，解释为什么以及何时大型人工神经网络能够很好地训练和推广，并使用该理论开发新的优化方法。新方法的实用性将在涉及语言、语音、生物序列和其他序列数据的应用中得到证明。该项目将涉及研究生和本科生的培训，项目负责人将提供针对机器学习社区以及其他使用机器学习工具的研究人员和工程师的教程。为了建立一个关于非凸优化在训练深度网络时为什么以及何时有效的理论，将采用经验自上而下和分析自下而上的方法。自上而下的方法将涉及对实践中使用的大规模深度模型的现象学分析，无论是在提供真实的数据时，还是在提供专门用于测试网络行为的数据时。自下而上的方法将涉及从越来越复杂的模型中进行精确的分析研究，从线性模型开始，非凸矩阵分解，通过线性神经网络，具有少量隐藏层的模型，最终达到更深，更复杂的网络。开发的理论旨在具有解释性和可操作性，并将用于推导新的优化方法和对架构的修改，以帮助优化和推广。一个特别重要的测试平台是递归神经网络。递归神经网络是功能强大的序列模型，在处理输入序列时保持状态，并用于序列数据。该奖项反映了NSF的法定使命，并通过使用基金会的知识价值和更广泛的影响审查标准进行评估，被认为值得支持。