Computational and Mathematical Studies of Complexity Reduction Methods for Deep Neural Networks and Applications

深度神经网络（DNN）已经成为推动人工智能最新进展的最先进的计算技术，在图像和语音识别任务以及玩复杂游戏（如围棋）方面超过了人类的表现。然而，深度网络通常会消耗数十亿的计算失误和千兆字节的模型和数据存储空间，这使得它们在移动和能源有限的平台（如蜂窝电话和电池驱动的汽车）上的部署面临挑战。该项目旨在发展复杂性降低方法的理论和算法，以在低计算预算下保持DNN的性能，实现加速和节省内存空间。该项目还通过自动架构搜索和选择来研究轻量级深度网络，以降低更高设计水平的复杂性。广泛的应用包括移动计算机视觉，疾病诊断和检测，人脸验证，以及无人机的监控和救援任务。该项目将积极吸引研究生参与，并通过教育和研究活动丰富他们的职业发展。要研究的方法包括：(1)使用低精度权重和激活函数训练深度网络（即所谓的量化）；(2)手工制作和自动化轻量级深度网络，通过变量分裂和量化进行训练。量化网络的训练涉及在离散约束下最小化高维不连续非凸目标，为此将分析新的粗梯度和加速技术（即所谓的混合）来引导下降并达到收敛。离散约束的可微处理，以及非光滑和组合结构的可微处理将得到充分发展。该项目的方法和最终算法将有助于信息技术，民用基础设施的优化，智能和高效的移动计算。该奖项反映了美国国家科学基金会的法定使命，并通过使用基金会的知识价值和更广泛的影响审查标准进行评估，被认为值得支持。

Deep neural networks (DNN) have become the state-of-the-art computing technology driving the recent advances in artificial intelligence, surpassing human performance on image and speech recognition tasks, and in playing complex games such as Go. However, deep networks typically consume billions of flops in computation and gigabytes of storage for model and data, rendering their deployment a challenge on mobile and energy limited platforms such as cellular phones and battery powered cars. The project aims to develop theory and algorithms for complexity reduction methods so as to maintain DNN's performance on low computational budget, achieving speed up and saving memory space. The project also studies light weight deep networks through automated architecture search and selection to reduce complexity at a higher design level. A broad range of applications include mobile computer vision, disease diagnosis and detection, face verification, as well as monitor and rescue missions by the drone. The project will actively involve graduate students and enrich their career development through both education and research activities. The approaches to be studied include (1) training of deep networks with low-precision weights and activation functions (so called quantization), (2) hand-crafted and automated lightweight deep networks, their training via variable splitting and their quantization. The training of quantized networks concerns with minimizing high dimensional discontinuous non-convex objectives under discrete constraints, for which novel coarse gradients and an accelerated technique (so called blending) will be analyzed to guide the descent and reach convergence. A differentiable treatment of discrete constraints, and of non-smooth and combinatorial structures will be fully developed. The methodologies and resulting algorithms from the project will contribute to information technology, optimization of civil infrastructure, smart and efficient mobile computing.This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.