Computational and Mathematical Studies of Compression and Distillation Methods for Deep Neural Networks and Applications

该项目将开发高效的深度学习架构，用于在资源有限的平台上部署人工智能算法，如移动计算和物联网。高性能的深度神经网络在计算中消耗数千亿次的失败，并在内存中存储数亿个参数。然而，在功率和内存资源有限的设备中，需要构建轻量级深度神经网络来保持重量级深度神经网络的性能水平。本项目旨在开发一种高效的基于搜索的架构压缩方法和一种新颖的教师-导师-学生（知识蒸馏）框架，在中间网络（导师）的帮助下，从最先进的重量级网络（教师）中提取智能轻量级网络（学生）。受益于该项目的现实世界应用包括移动电话上的视觉计算和自动驾驶，无人机的交付、监测和救援任务，以及移动医疗中的疾病检测和诊断。该项目将培训研究生，并为少数族裔服务机构的科学和工程本科学生提供丰富的数据科学课程。本项目将研究基于搜索的体系结构压缩的双网络协作方法，使低层网络权值和高层网络结构都得到有效优化。一个关键要素是将双层优化松弛为单层优化任务，并在搜索中采用可微代理函数近似的不可微决策。该项目将推进知识蒸馏方法，通过利用不同形状张量的相似性度量，以及图像分割中出现的多分辨率路径学习技术，将蒸馏扩展到教师网络的中间层。研究者还将制定简化的分类问题，用于数学分析和对蒸馏学习的理解。该奖项反映了美国国家科学基金会的法定使命，并通过使用基金会的知识价值和更广泛的影响审查标准进行评估，被认为值得支持。

This project will develop efficient deep learning architectures for the deployment of artificial intelligence algorithms on resource limited platforms, such as mobile computing and the internet of things. High performance deep neural networks consume hundreds of billions of flops in computation and store hundreds of millions of parameters in memory. However, devices with limited resources with respect to both power and memory call for constructions of lightweight deep neural networks to maintain the performance level of their heavyweight counterparts. This project aims to develop an efficient search-based architecture compression method and a novel teacher-tutor-student (knowledge distillation) framework to extract a smart lightweight network (student) from a state-of-the-art heavyweight network (teacher) with the help of an intermediate network (tutor). Real world applications benefitting from the project include visual computing on mobile phone and autonomous driving, the delivery, monitor and rescue missions by the drone, and disease detection and diagnosis in mobile health. The project will train graduate students and enrich data science curriculum for a diverse body of undergraduate students in science and engineering at minority serving institutions. The project will study a dual-network cooperation method for the search-based architecture compression so that the low level network weights and high level network structures are both optimized efficiently. A key element is a relaxation of bilevel optimization to a single level optimization task together with non-differentiable decision-making in the search approximated by a differentiable proxy function. The project will advance knowledge distillation methods, expanding distillation to intermediate layers of teacher networks by leveraging similarity measures on tensors of different shapes, and multi-resolution path learning techniques arising in image segmentations. The investigator will also formulate simplified classification problems for mathematical analysis and the understanding of distillation learning.This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.