CAREER: A Flexible Optimal Control Framework for Efficient Training of Deep Neural Networks

职业生涯：用于高效训练深度神经网络的灵活最优控制框架

• 批准号: 1751636
• 负责人: Lars Ruthotto
• 金额: $ 40万
• 依托单位: Emory University
• 依托单位国家: 美国
• 项目类别: Continuing Grant
• 财政年份: 2018
• 资助国家: 美国
• 起止时间: 2018-06-01 至 2024-05-31
• 项目状态: 已结题
• 来源: https://www.nsf.gov/awardsearch/showAward?AWD_ID=1751636&HistoricalAwards=false
• 关键词: CAREER; Flexible; Optimal; Control; Framework

One of the most transformative technologies of our time is deep learning, a form of machine learning that uses neural networks containing many hidden layers. Recent success has led to breakthroughs in applications such as speech and image recognition, nourishing public interest. However, more theoretical insight is needed to create a rigorous scientific basis for designing and training deep neural networks, increasing their scalability, and providing insight into their reasoning. This project develops a new mathematical framework that simplifies designing, training, and analyzing deep neural networks. Due to the broad applicability of deep learning, advances made in this project will benefit a wide range of technologies of high economic and societal impact, e.g., driverless cars, drug discoveries, and web searches. The mathematical theory supporting the new algorithms will increase the acceptance of deep learning for delicate tasks, e.g., cancer prediction and cyber-security. This project develops a new mathematical framework for deep learning based on the interpretation of deep learning as a dynamic optimal control problem involving nonlinear time-dependent differential equations. This interpretation offers a new way to analyze the successes and failures of deep learning. Advances in deep learning will be made by adapting the wide array of tools available for related optimal control problems, including numerical optimization, partial and ordinary differential equations, inverse problems theory, and parallel processing. Using these currently untapped resources provides rigorous new ways to design and train very deep neural networks that generalize well. Advances in optimal control will be achieved through new hybrid regularization methods, adaptive time discretizations, and large-scale splitting-based optimization.This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.

我们这个时代最具变革性的技术之一是深度学习，这是一种使用包含许多隐藏层的神经网络的机器学习形式。最近的成功导致了诸如语音和图像识别以及滋养公共利益之类的应用程序的突破。但是，需要更多的理论见解来创建严格的科学基础来设计和培训深层神经网络，提高其可扩展性并提供对其推理的见解。该项目开发了一个新的数学框架，可简化设计，培训和分析深层神经网络。由于深度学习的广泛适用性，该项目的进步将使广泛的经济和社会影响力的广泛技术受益，例如无人驾驶汽车，毒品发现和网络搜索。支持新算法的数学理论将增加对精致任务的深度学习的接受，例如癌症的预测和网络安全。该项目基于将深度学习作为一个动态最佳控制问题的解释开发出一种新的数学框架，以进行深度学习，涉及非线性时间依赖性的微分方程。这种解释提供了一种分析深度学习的成功和失败的新方法。深度学习的进步将通过调整可用于相关最佳控制问题的广泛工具，包括数值优化，部分和普通微分方程，反问题理论和并行处理。使用这些目前未开发的资源，可以提供严格的新方法来设计和训练非常深入的神经网络，从而可以很好地推广。最佳控制的进步将通过新的混合正规化方法，自适应时间离散化以及基于大规模拆分的优化实现。该奖项反映了NSF的法定任务，并被认为是值得通过基金会的知识分子优点和更广泛影响的审查标准通过评估来进行评估的。

项目成果

Layer-Parallel Training of Deep Residual Neural Networks

• DOI: 10.1137/19m1247620
• 发表时间: 2018-12
• 期刊: ArXiv
• 作者: Stefanie Günther;Lars Ruthotto;J. Schroder;E. Cyr;N. Gauger

A Neural Network Approach Applied to Multi-Agent Optimal Control

应用于多智能体最优控制的神经网络方法

• 发表时间: 2021
• 期刊: European Control Conference
• 作者: Onken, D;Nurbekyan, L;Li, Xingjian;Wu Fung, S;Osher, S;Ruthotto, L
• 通讯作者: Ruthotto, L

PNKH-B: A Projected Newton--Krylov Method for Large-Scale Bound-Constrained Optimization

PNKH-B：用于大规模边界约束优化的投影 Newton-Krylov 方法

• DOI: 10.1137/20m1341428
• 发表时间: 2021
• 期刊: SIAM Journal on Scientific Computing
• 影响因子: 3.1
• 作者: Kan, Kelvin;Fung, Samy Wu;Ruthotto, Lars
• 通讯作者: Ruthotto, Lars

A machine learning framework for solving high-dimensional mean field game and mean field control problems

• DOI: 10.1073/pnas.1922204117
• 发表时间: 2020-04-28
• 期刊: PROCEEDINGS OF THE NATIONAL ACADEMY OF SCIENCES OF THE UNITED STATES OF AMERICA
• 影响因子: 11.1
• 作者: Ruthotto, Lars;Osher, Stanley J.;Fung, Samy Wu
• 通讯作者: Fung, Samy Wu

A Tensor SVD-based Classification Algorithm Applied to fMRI Data

• DOI: 10.1137/21s1456522
• 发表时间: 2021-10
• 期刊: ArXiv
• 作者: K. Keegan;T. Vishwanath;Yihua Xu