CAREER: A Flexible Optimal Control Framework for Efficient Training of Deep Neural Networks

我们这个时代最具变革性的技术之一是深度学习，这是一种使用包含许多隐藏层的神经网络的机器学习形式。最近的成功导致了语音和图像识别等应用的突破，滋养了公众的兴趣。然而，需要更多的理论见解来为设计和训练深度神经网络创造严格的科学基础，增加它们的可扩展性，并为它们的推理提供洞察。该项目开发了一种新的数学框架，简化了深度神经网络的设计、训练和分析。由于深度学习的广泛适用性，该项目取得的进展将使具有高经济和社会影响的广泛技术受益，例如无人驾驶汽车、药物发现和网络搜索。支持新算法的数学理论将增加对精细任务的深度学习的接受度，例如癌症预测和网络安全。这个项目开发了一种新的深度学习的数学框架，其基础是深度学习被解释为一个涉及非线性时变微分方程的动态最优控制问题。这一解读为分析深度学习的成败提供了一种新的视角。通过采用可用于相关最优控制问题的广泛工具，包括数值优化、偏微分方程组和常微分方程组、反问题理论和并行处理，将在深度学习方面取得进展。利用这些目前尚未开发的资源，可以提供严格的新方法来设计和训练非常深入的神经网络，使其具有良好的泛化能力。最优控制方面的进展将通过新的混合正则化方法、自适应时间离散化和基于大规模分裂的优化实现。该奖项反映了NSF的法定使命，并通过使用基金会的智力优势和更广泛的影响审查标准进行评估，被认为值得支持。

One of the most transformative technologies of our time is deep learning, a form of machine learning that uses neural networks containing many hidden layers. Recent success has led to breakthroughs in applications such as speech and image recognition, nourishing public interest. However, more theoretical insight is needed to create a rigorous scientific basis for designing and training deep neural networks, increasing their scalability, and providing insight into their reasoning. This project develops a new mathematical framework that simplifies designing, training, and analyzing deep neural networks. Due to the broad applicability of deep learning, advances made in this project will benefit a wide range of technologies of high economic and societal impact, e.g., driverless cars, drug discoveries, and web searches. The mathematical theory supporting the new algorithms will increase the acceptance of deep learning for delicate tasks, e.g., cancer prediction and cyber-security. This project develops a new mathematical framework for deep learning based on the interpretation of deep learning as a dynamic optimal control problem involving nonlinear time-dependent differential equations. This interpretation offers a new way to analyze the successes and failures of deep learning. Advances in deep learning will be made by adapting the wide array of tools available for related optimal control problems, including numerical optimization, partial and ordinary differential equations, inverse problems theory, and parallel processing. Using these currently untapped resources provides rigorous new ways to design and train very deep neural networks that generalize well. Advances in optimal control will be achieved through new hybrid regularization methods, adaptive time discretizations, and large-scale splitting-based optimization.This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.