CAREER: A Flexible Optimal Control Framework for Efficient Training of Deep Neural Networks

职业生涯：用于高效训练深度神经网络的灵活最优控制框架

• 批准号: 1751636
• 负责人: Lars Ruthotto
• 金额: $ 40万
• 依托单位: Emory University
• 依托单位国家: 美国
• 项目类别: Continuing Grant
• 财政年份: 2018
• 资助国家: 美国
• 起止时间: 2018-06-01 至 2024-05-31
• 项目状态: 已结题
• 来源: https://www.nsf.gov/awardsearch/showAward?AWD_ID=1751636&HistoricalAwards=false
• 关键词: CAREER; Flexible; Optimal; Control; Framework

One of the most transformative technologies of our time is deep learning, a form of machine learning that uses neural networks containing many hidden layers. Recent success has led to breakthroughs in applications such as speech and image recognition, nourishing public interest. However, more theoretical insight is needed to create a rigorous scientific basis for designing and training deep neural networks, increasing their scalability, and providing insight into their reasoning. This project develops a new mathematical framework that simplifies designing, training, and analyzing deep neural networks. Due to the broad applicability of deep learning, advances made in this project will benefit a wide range of technologies of high economic and societal impact, e.g., driverless cars, drug discoveries, and web searches. The mathematical theory supporting the new algorithms will increase the acceptance of deep learning for delicate tasks, e.g., cancer prediction and cyber-security. This project develops a new mathematical framework for deep learning based on the interpretation of deep learning as a dynamic optimal control problem involving nonlinear time-dependent differential equations. This interpretation offers a new way to analyze the successes and failures of deep learning. Advances in deep learning will be made by adapting the wide array of tools available for related optimal control problems, including numerical optimization, partial and ordinary differential equations, inverse problems theory, and parallel processing. Using these currently untapped resources provides rigorous new ways to design and train very deep neural networks that generalize well. Advances in optimal control will be achieved through new hybrid regularization methods, adaptive time discretizations, and large-scale splitting-based optimization.This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.

我们这个时代最具变革性的技术之一是深度学习，这是一种使用包含许多隐藏层的神经网络的机器学习形式。最近的成功导致了语音和图像识别等应用的突破，培养了公众的兴趣。然而，需要更多的理论见解来为设计和训练深度神经网络创建严格的科学基础，提高其可扩展性，并提供对其推理的洞察力。该项目开发了一个新的数学框架，简化了深度神经网络的设计、训练和分析。由于深度学习的广泛适用性，该项目取得的进展将使广泛的具有高经济和社会影响力的技术受益，例如，无人驾驶汽车、药物发现和网络搜索。支持新算法的数学理论将提高深度学习在精细任务中的接受度，例如，癌症预测和网络安全该项目基于深度学习作为涉及非线性时变微分方程的动态最优控制问题的解释，为深度学习开发了一个新的数学框架。这种解释为分析深度学习的成功和失败提供了一种新的方法。深度学习的进步将通过适应相关最优控制问题的广泛工具来实现，包括数值优化，偏微分方程和常微分方程，反问题理论和并行处理。使用这些目前尚未开发的资源提供了严格的新方法来设计和训练非常深入的神经网络，这些神经网络具有良好的泛化能力。通过新的混合正则化方法、自适应时间离散化和大规模基于分裂的优化，将实现最优控制的进步。该奖项反映了NSF的法定使命，并通过使用基金会的智力价值和更广泛的影响审查标准进行评估，被认为值得支持。

项目成果

slimTrain---A Stochastic Approximation Method for Training Separable Deep Neural Networks

slimTrain---一种训练可分离深度神经网络的随机逼近方法

• DOI: 10.1137/21m1452512
• 发表时间: 2022
• 期刊: SIAM Journal on Scientific Computing
• 影响因子: 3.1
• 作者: Newman, Elizabeth;Chung, Julianne;Chung, Matthias;Ruthotto, Lars
• 通讯作者: Ruthotto, Lars

Layer-Parallel Training of Deep Residual Neural Networks

• DOI: 10.1137/19m1247620
• 发表时间: 2018-12
• 期刊: ArXiv
• 作者: Stefanie Günther;Lars Ruthotto;J. Schroder;E. Cyr;N. Gauger

IMEXnet: A Forward Stable Deep Neural Network

• 发表时间: 2019-03
• 期刊: ArXiv
• 作者: E. Haber;Keegan Lensink;Eran Treister;Lars Ruthotto

A Neural Network Approach Applied to Multi-Agent Optimal Control

应用于多智能体最优控制的神经网络方法

• 发表时间: 2021
• 期刊: European Control Conference
• 作者: Onken, D;Nurbekyan, L;Li, Xingjian;Wu Fung, S;Osher, S;Ruthotto, L
• 通讯作者: Ruthotto, L

PNKH-B: A Projected Newton--Krylov Method for Large-Scale Bound-Constrained Optimization

PNKH-B：用于大规模边界约束优化的投影 Newton-Krylov 方法

• DOI: 10.1137/20m1341428
• 发表时间: 2021
• 期刊: SIAM Journal on Scientific Computing
• 影响因子: 3.1
• 作者: Kan, Kelvin;Fung, Samy Wu;Ruthotto, Lars
• 通讯作者: Ruthotto, Lars