Collaborative Research: Algebraic Framework of Compositional Functions for New Structure, Training, and Explainability of Deep Learning

合作研究：深度学习新结构、训练和可解释性的组合函数代数框架

• 批准号: 2134235
• 负责人: Qi Gong
• 金额: $ 45.85万
• 依托单位: University of California-Santa Cruz
• 依托单位国家: 美国
• 项目类别: Standard Grant
• 财政年份: 2022
• 资助国家: 美国
• 起止时间: 2022-01-01 至 2024-12-31
• 项目状态: 已结题
• 来源: https://www.nsf.gov/awardsearch/showAward?AWD_ID=2134235&HistoricalAwards=false
• 关键词: Collaborative; Research; Algebraic; Framework; Compositional

Deep learning is a method of machine learning inspired by the human brain. Data is fed to a multi-layered (deep) network of trainable 'neurons' and the network is then trained to model complex relations and processes. Deep learning has had impressive success in applications such as image recognition and natural language processing. And yet, there are few theoretical guarantees to the method, to provide assurances in regards to performance features such as error and to explain its success. This is an impediment to broader application of deep learning, as many potential applications require guarantees for safety, reliability, and accuracy. This project pursues a solid mathematical foundation for a better understanding of the explainability of deep learning, to enable more efficient neural network design and training algorithms that benefit a wide range of applications. The project also includes a significant educational component that is designed to foster interdisciplinary education by engaging undergraduate and graduate students from the investigators' departments (Applied Mathematics, Electrical Engineering, Computer Science, Mathematics) in the proposed multidisciplinary research. The project includes plans to promote diversity, equity and inclusion in STEM education at the University of California Santa Cruz and the University of Texas at San Antonio, which are both Hispanic Serving Institutions.The overarching goal of this project is to develop a unified algebraic framework and approximation theory for deep neural networks so that the framework is applicable to a wide spectrum of problems including regression, solving differential equations, designing optimal feedback control, and computer vision. The proposed research is motivated by the fact that most complicated and high dimensional input-output relations in real-world applications can be represented as compositions of simple low-dimensional functions. Thus, compositional functions, including deep neural networks, serve as a natural way to describe complex high dimensional functions. Representing compositional functions as layered acyclic graphs, the project will explore the compositional features of the problems to be solved by machine learning; study the error propagation in layered acyclic graphs; and investigate the interconnection between the compositional features and the fundamental issues of machine learning, such as the error bounds in universal approximation, deep neural network design and training, and validation and explainability. The algebraic framework, approximation theory, and computational algorithms to be developed in this research project should advance the design, training, and mathematical foundations of deep learning. They seek also to be directly applicable to a wide spectrum of applications including feedback control design and computer vision, which are included in this project, as well as other important machine learning applications, such as regression and data-driven modeling of dynamical systems.This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.

深度学习是一种受人脑启发的机器学习方法。数据被馈送到可训练的“神经元”的多层（深度）网络，然后网络被训练以模拟复杂的关系和过程。深度学习在图像识别和自然语言处理等应用中取得了令人印象深刻的成功。然而，该方法几乎没有理论保证，无法保证错误等性能特征并解释其成功。这阻碍了深度学习的广泛应用，因为许多潜在的应用都需要安全性、可靠性和准确性的保证。该项目追求坚实的数学基础，以更好地理解深度学习的可解释性，从而实现更有效的神经网络设计和训练算法，使广泛的应用受益。该项目还包括一个重要的教育组成部分，旨在促进跨学科教育，让研究人员部门（应用数学，电气工程，计算机科学，数学）的本科生和研究生参与拟议的多学科研究。该项目包括促进加州大学圣克鲁斯分校和德克萨斯大学圣安东尼奥分校STEM教育的多样性、公平性和包容性的计划，这两所大学都是西班牙裔服务机构。该项目的首要目标是为深度神经网络开发统一的代数框架和近似理论，以便该框架适用于包括回归、解微分方程、设计最优反馈控制和计算机视觉。所提出的研究的动机是，在现实世界中的应用程序中，最复杂的和高维的输入输出关系可以表示为简单的低维函数的组合。因此，包括深度神经网络在内的组合函数是描述复杂高维函数的自然方式。将组合函数表示为分层非循环图，该项目将探索机器学习要解决的问题的组合特征;研究分层非循环图中的错误传播;并研究组合特征与机器学习的基本问题之间的相互联系，例如通用近似中的误差界，深度神经网络设计和训练，以及验证和可解释性。在这个研究项目中开发的代数框架、近似理论和计算算法应该推进深度学习的设计、训练和数学基础。他们还寻求直接适用于广泛的应用，包括反馈控制设计和计算机视觉，这些都包括在这个项目中，以及其他重要的机器学习应用，例如回归和数据-该奖项反映了NSF的法定使命，并通过使用基金会的知识价值和更广泛的影响审查进行评估，被认为值得支持的搜索.

项目成果

Compositional Features and Neural Network Complexity in Deep Learning

深度学习中的组成特征和神经网络复杂性

• 发表时间: 2022
• 期刊: 25th International Symposium on Mathematical Theory of Networks and Systems (MTNS
• 作者: Gong, Qi;Kang, Wei
• 通讯作者: Kang, Wei

An Actor Critic Method for Free Terminal Time Optimal Control

自由终端时间最优控制的Actor批评方法

• 发表时间: 2023
• 期刊: The 12th IFAC Symposium on Nonlinear Control Systems (NOLCOS
• 作者: Burton, Evan;Nakamura-Zimmerer, Tenavi;Gong, Qi;Kang, Wei
• 通讯作者: Kang, Wei

The Observability in Unobservable Systems

不可观测系统的可观测性

• DOI: 10.1109/icca54724.2022.9831888
• 发表时间: 2022
• 期刊: Italy
• 作者: Kang, Wei;Xu, Liang;Zhou, Hong
• 通讯作者: Zhou, Hong

Approximation of compositional functions with ReLU neural networks

使用 ReLU 神经网络逼近复合函数

• 发表时间: 2023
• 期刊: Systems control letters
• 作者: Gong, Q.;Kang, W.;Fahroo, F.
• 通讯作者: Fahroo, F.

Neural Network Optimal Feedback Control With Guaranteed Local Stability

保证局部稳定性的神经网络最优反馈控制

• DOI: 10.1109/ojcsys.2022.3205863
• 发表时间: 2022
• 期刊: IEEE Open Journal of Control Systems
• 作者: Nakamura-Zimmerer, Tenavi;Gong, Qi;Kang, Wei
• 通讯作者: Kang, Wei