Computational and Mathematical Studies of Compression and Distillation Methods for Deep Neural Networks and Applications

深度神经网络压缩和蒸馏方法的计算和数学研究及应用

• 批准号: 2151235
• 负责人: Jack Xin
• 金额: $ 30万
• 依托单位: University of California-Irvine
• 依托单位国家: 美国
• 项目类别: Continuing Grant
• 财政年份: 2022
• 资助国家: 美国
• 起止时间: 2022-09-01 至 2025-08-31
• 项目状态: 未结题
• 来源: https://www.nsf.gov/awardsearch/showAward?AWD_ID=2151235&HistoricalAwards=false
• 关键词: Computational; Mathematical; Studies; Compression; Distillation

This project will develop efficient deep learning architectures for the deployment of artificial intelligence algorithms on resource limited platforms, such as mobile computing and the internet of things. High performance deep neural networks consume hundreds of billions of flops in computation and store hundreds of millions of parameters in memory. However, devices with limited resources with respect to both power and memory call for constructions of lightweight deep neural networks to maintain the performance level of their heavyweight counterparts. This project aims to develop an efficient search-based architecture compression method and a novel teacher-tutor-student (knowledge distillation) framework to extract a smart lightweight network (student) from a state-of-the-art heavyweight network (teacher) with the help of an intermediate network (tutor). Real world applications benefitting from the project include visual computing on mobile phone and autonomous driving, the delivery, monitor and rescue missions by the drone, and disease detection and diagnosis in mobile health. The project will train graduate students and enrich data science curriculum for a diverse body of undergraduate students in science and engineering at minority serving institutions. The project will study a dual-network cooperation method for the search-based architecture compression so that the low level network weights and high level network structures are both optimized efficiently. A key element is a relaxation of bilevel optimization to a single level optimization task together with non-differentiable decision-making in the search approximated by a differentiable proxy function. The project will advance knowledge distillation methods, expanding distillation to intermediate layers of teacher networks by leveraging similarity measures on tensors of different shapes, and multi-resolution path learning techniques arising in image segmentations. The investigator will also formulate simplified classification problems for mathematical analysis and the understanding of distillation learning.This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.

该项目将开发高效的深度学习架构，用于在移动计算和物联网等资源有限的平台上部署人工智能算法。高性能深度神经网络在计算上消耗数千亿次浮点运算，在内存中存储数亿个参数。然而，在电力和存储资源有限的设备上，需要构建轻量级深度神经网络来保持其重量级设备的性能水平。该项目旨在开发一种高效的基于搜索的体系结构压缩方法和一种新颖的教师-教师-学生(知识蒸馏)框架，以在中间网络(导师)的帮助下从最先进的重量级网络(教师)中提取智能轻量级网络(学生)。受益于该项目的现实世界应用包括移动电话和自动驾驶的视觉计算，无人机的交付、监测和救援任务，以及移动医疗中的疾病检测和诊断。该项目将为少数民族服务机构的理工科本科生培养研究生，并丰富数据科学课程。该项目将研究一种基于搜索的双网络协作结构压缩方法，使低层网络权重和高层网络结构都得到有效优化。一个关键因素是将双层优化放宽到单层优化任务，并用可微代理函数逼近搜索中的不可微决策。该项目将推进知识蒸馏方法，通过利用不同形状张量的相似性度量和图像分割中出现的多分辨率路径学习技术，将蒸馏扩展到教师网络的中间层。研究人员还将为数学分析和对蒸馏学习的理解制定简化的分类问题。这一奖项反映了NSF的法定使命，并通过使用基金会的智力优势和更广泛的影响审查标准进行评估，被认为值得支持。

项目成果

Feature Affinity Assisted Knowledge Distillation and Quantization of Deep Neural Networks on Label-Free Data

• DOI: 10.1109/access.2023.3297890
• 发表时间: 2023-02
• 期刊: IEEE Access
• 影响因子: 3.9
• 作者: Zhijian Li;Biao Yang;Penghang Yin;Y. Qi;J. Xin

Convergence of Hyperbolic Neural Networks Under Riemannian Stochastic Gradient Descent

黎曼随机梯度下降下双曲神经网络的收敛性

• DOI: 10.1007/s42967-023-00302-9
• 发表时间: 2023
• 期刊: Communications on Applied Mathematics and Computation
• 影响因子: 1.6
• 作者: Whiting, Wes;Wang, Bao;Xin, Jack
• 通讯作者: Xin, Jack

Weighted Anisotropic–Isotropic Total Variation for Poisson Denoising

• DOI: 10.1109/icip49359.2023.10222230
• 发表时间: 2023-07
• 期刊: 2023 IEEE International Conference on Image Processing (ICIP)
• 作者: Kevin Bui;Yifei Lou;Fredrick Park;J. Xin

Difference of anisotropic and isotropic TV for segmentation under blur and Poisson noise

• DOI: 10.3389/fcomp.2023.1131317
• 发表时间: 2023-01
• 作者: Kevin Bui;Yifei Lou;Fredrick Park;J. Xin