AF: RI: Medium: Collaborative Research: Understanding and Improving Optimization in Deep and Recurrent Networks

AF：RI：中：协作研究：理解和改进深度和循环网络的优化

基本信息

批准号：
1764032
负责人：
Nathan Srebro
金额：
$ 54.11万
依托单位：
Toyota Technological Institute at Chicago
依托单位国家：
美国
项目类别：
Standard Grant
财政年份：
2018
资助国家：
美国
起止时间：
2018-08-01 至 2024-09-30
项目状态：
已结题

来源：
https://www.nsf.gov/awardsearch/showAward?AWD_ID=1764032&HistoricalAwards=false
关键词：
AF RI Medium Collaborative Research

项目摘要

Machine learning using deep neural networks has recently demonstrated broad empirical success. Despite this success, the optimization procedures that fit deep neural networks to data are still poorly understood. Besides playing a crucial role in fitting deep neural networks to data, optimization also strongly affects the model's ability to generalize from training examples to unseen data. This project will establish a working theory for why and when large artificial neural networks train and generalize well, and use this theory to develop new optimization methods. The utility of the new methods will be demonstrated in applications involving language, speech, biological sequences and other sequence data. The project will involve training of graduate and undergraduate students, and the project leaders will offer tutorials aimed at both the machine learning community, and other researchers and engineers using machine learning tools. In order to establish a theory of why and when non-convex optimization works well when training deep networks, both empirical top-down and analytic bottom-up approaches will be pursued. The top-down approach will involve phenomenological analysis of large scale deep models used in practice, both when presented with real data, and when presented with data specifically crafted to test the behavior of the network. The bottom-up approach will involve precise analytic investigation from increasingly more complex models, starting with linear models, and non-convex matrix factorization, progressing through linear neural networks, models with a small number of hidden layers, and eventually reaching deeper and more complex networks. The theory developed aims to be both explanatory and actionable, and will be used to derive new optimization methods and modifications to architectures that aid in optimization and generalization. A particularly important testbed is the case of recurrent neural networks. Recurrent neural networks are powerful sequence models that maintain state as they process an input sequence and are used for sequence data. Particularly challenging to optimize, recurrent neural networks still leave much room for a stronger principled understanding, which the project aims to provide.This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.

使用深神经网络的机器学习最近证明了广泛的经验成功。尽管取得了成功，但适合深度神经网络数据的优化程序仍然对数据的理解仍然很差。除了在将深度神经网络拟合到数据中起着至关重要的作用外，优化还强烈影响该模型从训练示例到看不见的数据的概括能力。该项目将建立一个工作理论，说明为什么以及何时大型人工神经网络训练和概括，并使用该理论来开发新的优化方法。新方法的实用性将在涉及语言，语音，生物序列和其他序列数据的应用中证明。该项目将涉及培训研究生和本科生，项目负责人将提供针对机器学习社区的教程，以及使用机器学习工具的其他研究人员和工程师。为了建立一个理论以及何时在训练深层网络时非凸优化效果很好的理论，将采用经验性的自上而下和分析自下而上的方法。自上而下的方法将涉及对实践中使用的大规模深模型的现象学分析，无论是在带有真实数据时，都将介绍专门用于测试网络行为的数据时。自下而上的方法将涉及越来越复杂的模型（从线性模型开始）和非凸矩阵分解的精确分析研究，通过线性神经网络进行，具有少量隐藏层的模型，并最终达到更深入且更复杂的网络。所开发的理论旨在既是解释性又可行，并将用于为有助于优化和概括的体系结构提供新的优化方法和修改。一个特别重要的测试是复发性神经网络的情况。复发性神经网络是强大的序列模型，它们在处理输入序列并用于序列数据时保持状态。优化反复的神经网络特别具有挑战性，仍留出很大的原则性理解的空间，该项目旨在提供该奖项。该奖项反映了NSF的法定任务，并被认为是值得通过基金会的知识分子和更广泛影响的评估评估标准来通过评估来获得支持的。

项目成果

期刊论文数量（33）

专著数量（0）

科研奖励数量（0）

会议论文数量（0）

专利数量（0）

Characterizing Implicit Bias in Terms of Optimization Geometry

DOI：
发表时间：
2018-02
期刊：
ArXiv
影响因子：
0
作者：
Suriya Gunasekar;Jason D. Lee;Daniel Soudry;N. Srebro
通讯作者：
Suriya Gunasekar;Jason D. Lee;Daniel Soudry;N. Srebro

On the Implicit Bias of Initialization Shape: Beyond Infinitesimal Mirror Descent

DOI：
发表时间：
2021-02
期刊：
ArXiv
影响因子：
0
作者：
Shahar Azulay;E. Moroshko;M. S. Nacson;Blake E. Woodworth;N. Srebro;A. Globerson;Daniel Soudry
通讯作者：
Shahar Azulay;E. Moroshko;M. S. Nacson;Blake E. Woodworth;N. Srebro;A. Globerson;Daniel Soudry

Pessimism for Offline Linear Contextual Bandits using Confidence Sets

使用置信集对离线线性上下文强盗的悲观态度