Algorithms and Theory for Compressing Deep Neural Networks

深度神经网络(DNN)一直是近年来人工智能(AI)技术进步的主要驱动力，在交通、公共安全、娱乐、医疗保健和其他公共生活领域对社会产生了深远的影响。人工智能对我们日常生活产生更广泛影响的最大障碍之一是DNN在部署时通常会产生巨大的功耗。该项目的目的是开发DNN压缩的数学和计算方法，以实现AI系统在智能手机等低功耗预算的移动平台上的快速高效部署。这项工作的成果将有各种应用，包括视频安全系统，自动驾驶，智能机器人，和人脸识别。该项目将涉及研究生的培训、数据科学课程的开发以及与业界的合作。PI计划(1)开发和分析具有偏向一阶预言的粗梯度算法，用于包括基于变压器的网络在内的各种神经结构的离散化；(2)开发和分析基于阈值的高效网络压缩算法，通过对平衡和非平衡数据的结构化稀疏性来压缩网络；(3)研究压缩DNN的模型容量并建立通用的有限样本表达理论。这项拟议的研究还将探索粗梯度算法在其他具有离散值损失函数和离散优化高级知识的机器学习问题中的应用。这一奖项反映了NSF的法定使命，并通过使用基金会的智力优势和更广泛的影响审查标准进行评估，被认为值得支持。

Deep neural networks (DNNs) have been the main driving force for recent advancements in artificial intelligence (AI) technology, profoundly impacting society in the areas of transportation, public safety, entertainment, health care, and other areas of public life. One of the biggest obstacles to AI's even broader impact on our daily lives is the typically enormous power consumption of DNNs upon deployment. The aim of this project is to develop mathematical and computational approaches for DNN compression to realize the fast and efficient deployment of AI systems on mobile platforms with low-power budgets such as smartphones. Results of this work will have a variety of applications which include video security systems, autopilot, smart robots, and face identification. The project will involve training of graduate students, development of data science courses, as well as collaboration with industry. The PI plans to (1) develop and analyze coarse gradient algorithms, featuring a biased first-order oracle, for the discretization of various neural architectures including transformer-based networks; (2) develop and analyze efficient thresholding-based algorithms for compressing networks via structured sparsity on both balanced and unbalanced data; (3) investigate the model capacity of compressed DNNs and establish universal finite-sample expressivity theory. The proposed research will also explore the applications of coarse gradient algorithms to other machine learning problems with discrete-valued loss functions and advance knowledge in discrete optimization.This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.