Computational and Mathematical Studies of Complexity Reduction Methods for Deep Neural Networks and Applications

深度神经网络复杂度降低方法的计算和数学研究及应用

• 批准号: 1854434
• 负责人: Jack Xin
• 金额: $ 30万
• 依托单位: University of California-Irvine
• 依托单位国家: 美国
• 项目类别: Standard Grant
• 财政年份: 2019
• 资助国家: 美国
• 起止时间: 2019-07-15 至 2023-06-30
• 项目状态: 已结题
• 来源: https://www.nsf.gov/awardsearch/showAward?AWD_ID=1854434&HistoricalAwards=false
• 关键词: Computational; Mathematical; Studies; Complexity; Reduction

Deep neural networks (DNN) have become the state-of-the-art computing technology driving the recent advances in artificial intelligence, surpassing human performance on image and speech recognition tasks, and in playing complex games such as Go. However, deep networks typically consume billions of flops in computation and gigabytes of storage for model and data, rendering their deployment a challenge on mobile and energy limited platforms such as cellular phones and battery powered cars. The project aims to develop theory and algorithms for complexity reduction methods so as to maintain DNN's performance on low computational budget, achieving speed up and saving memory space. The project also studies light weight deep networks through automated architecture search and selection to reduce complexity at a higher design level. A broad range of applications include mobile computer vision, disease diagnosis and detection, face verification, as well as monitor and rescue missions by the drone. The project will actively involve graduate students and enrich their career development through both education and research activities. The approaches to be studied include (1) training of deep networks with low-precision weights and activation functions (so called quantization), (2) hand-crafted and automated lightweight deep networks, their training via variable splitting and their quantization. The training of quantized networks concerns with minimizing high dimensional discontinuous non-convex objectives under discrete constraints, for which novel coarse gradients and an accelerated technique (so called blending) will be analyzed to guide the descent and reach convergence. A differentiable treatment of discrete constraints, and of non-smooth and combinatorial structures will be fully developed. The methodologies and resulting algorithms from the project will contribute to information technology, optimization of civil infrastructure, smart and efficient mobile computing.This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.

深度神经网络（DNN）已经成为最先进的计算技术，推动了人工智能的最新进展，在图像和语音识别任务以及围棋等复杂游戏中超越了人类的表现。然而，深度网络通常在计算中消耗数十亿次失败以及用于模型和数据的千兆字节的存储，使得它们的部署在移动的和能量有限的平台（诸如蜂窝电话和电池供电的汽车）上成为挑战。该项目旨在开发降低复杂性方法的理论和算法，以便在低计算预算下保持DNN的性能，实现加速和节省内存空间。 该项目还通过自动架构搜索和选择来研究轻量级深度网络，以降低更高设计级别的复杂性。广泛的应用包括移动的计算机视觉、疾病诊断和检测、人脸验证以及无人机的监控和救援任务。该项目将通过教育和研究活动积极吸引研究生参与，丰富他们的职业发展。要研究的方法包括（1）使用低精度权重和激活函数（所谓的量化）训练深度网络，（2）手工制作和自动化的轻量级深度网络，通过变量拆分和量化进行训练。量化网络的训练涉及在离散约束下最小化高维不连续非凸目标，为此将分析新的粗梯度和加速技术（所谓的混合）以指导下降并达到收敛。一个可微处理离散约束，非光滑和组合结构将得到充分发展。该项目的方法和算法将有助于信息技术、民用基础设施的优化、智能和高效的移动的计算。该奖项反映了NSF的法定使命，并通过使用基金会的知识价值和更广泛的影响审查标准进行评估，被认为值得支持。

项目成果

Two-Grid Based Adaptive Proper Orthogonal Decomposition Method for Time Dependent Partial Differential Equations

基于二网格的时变偏微分方程自适应原正交分解方法

• DOI: 10.1007/s10915-020-01288-9
• 发表时间: 2020-08
• 期刊: Journal of Scientific Computing
• 影响因子: 2.5
• 作者: Dai Xiaoying;Kuang Xiong;Xin Jack;Zhou Aihui
• 通讯作者: Zhou Aihui

Searching Intrinsic Dimensions of Vision Transformers

寻找视觉变压器的内在维度

• DOI: 10.17758/heaig10.h0622602
• 发表时间: 2022
• 期刊: The 20th International Conference on Innovations in Engineering and Sciences
• 作者: Xue, Fanghui;Yang, Biao;Qi, Yingyong;Xin, Jack
• 通讯作者: Xin, Jack

A Channel-Pruned and Weight-Binarized Convolutional Neural Network for Keyword Spotting

• DOI: 10.1007/978-3-030-38364-0_22
• 发表时间: 2019-09
• 期刊: ArXiv
• 作者: J. Lyu;S. Sheen

Global convergence and geometric characterization of slow to fast weight evolution in neural network training for classifying linearly non-separable data

用于分类线性不可分离数据的神经网络训练中从慢到快权重演化的全局收敛和几何表征

• DOI: 10.3934/ipi.2020077
• 发表时间: 2021
• 期刊: Inverse Problems & Imaging
• 影响因子: 1.3
• 作者: Long, Ziang;Yin, Penghang;Xin, Jack
• 通讯作者: Xin, Jack

Synchronized Front Propagation and Delayed Flame Quenching in Strain G-Equation and Time-Periodic Cellular Flows

应变 G 方程和时间周期细胞流中的同步前沿传播和延迟火焰淬火

• 发表时间: 2023
• 期刊: Minimax theory and its applications
• 影响因子: 0.7
• 作者: Liu, Yu-Yu;Xin, Jack
• 通讯作者: Xin, Jack