Algorithms and Theory for Compressing Deep Neural Networks

深度神经网络（dnn）一直是人工智能（AI）技术进步的主要推动力，在交通、公共安全、娱乐、医疗保健和其他公共生活领域深刻地影响着社会。人工智能对我们日常生活产生更广泛影响的最大障碍之一是深度神经网络在部署时的巨大功耗。该项目的目的是开发DNN压缩的数学和计算方法，以实现在智能手机等低功耗预算的移动平台上快速有效地部署AI系统。这项工作的结果将有各种各样的应用，包括视频安全系统，自动驾驶，智能机器人和面部识别。该项目将包括培训研究生、开发数据科学课程，以及与工业界合作。PI计划(1)开发和分析粗梯度算法，具有偏一阶预言，用于各种神经架构的离散化，包括基于变压器的网络；(2)开发和分析基于阈值的高效算法，通过结构化稀疏性对平衡和非平衡数据进行压缩；(3)研究压缩深度神经网络的模型能力，建立通用有限样本表达性理论。提出的研究还将探索粗梯度算法在其他具有离散值损失函数的机器学习问题中的应用，并推进离散优化方面的知识。该奖项反映了美国国家科学基金会的法定使命，并通过使用基金会的知识价值和更广泛的影响审查标准进行评估，被认为值得支持。

Deep neural networks (DNNs) have been the main driving force for recent advancements in artificial intelligence (AI) technology, profoundly impacting society in the areas of transportation, public safety, entertainment, health care, and other areas of public life. One of the biggest obstacles to AI's even broader impact on our daily lives is the typically enormous power consumption of DNNs upon deployment. The aim of this project is to develop mathematical and computational approaches for DNN compression to realize the fast and efficient deployment of AI systems on mobile platforms with low-power budgets such as smartphones. Results of this work will have a variety of applications which include video security systems, autopilot, smart robots, and face identification. The project will involve training of graduate students, development of data science courses, as well as collaboration with industry. The PI plans to (1) develop and analyze coarse gradient algorithms, featuring a biased first-order oracle, for the discretization of various neural architectures including transformer-based networks; (2) develop and analyze efficient thresholding-based algorithms for compressing networks via structured sparsity on both balanced and unbalanced data; (3) investigate the model capacity of compressed DNNs and establish universal finite-sample expressivity theory. The proposed research will also explore the applications of coarse gradient algorithms to other machine learning problems with discrete-valued loss functions and advance knowledge in discrete optimization.This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.