CNS Core: Small: Interpretable Multi-Modal Neural Network Pruning for Edge Devices

CNS 核心：小型：边缘设备的可解释多模态神经网络修剪

• 批准号: 1908658
• 负责人: Ziliang Zong
• 金额: $ 50万
• 依托单位: Texas State University - San Marcos
• 依托单位国家: 美国
• 项目类别: Standard Grant
• 财政年份: 2019
• 资助国家: 美国
• 起止时间: 2019-10-01 至 2023-09-30
• 项目状态: 已结题
• 来源: https://www.nsf.gov/awardsearch/showAward?AWD_ID=1908658&HistoricalAwards=false
• 关键词: CNS; Core; Small; Interpretable; Multi

Deep artificial neural network technology has been widely used to solve many challenging tasks in computer vision, natural language processing, speech recognition, and more. Most of today's deep learning algorithms are designed for high performance servers and running in the cloud. As the edge devices (e.g., mobile phones and smart watches) become more capable and the advantages of on-device artificial intelligence (AI) (e.g. protecting privacy, working without a network, processing data locally in real-time) become more evident, bringing AI to the edge will be inevitable. However, the limited resources (e.g., computation, memory, and battery) of edge devices bring a whole new level of challenges: (1) On-device AI must keep the model size small without sacrificing accuracy; (2) On-device AI must keep the power usage low; (3) Future on-device AI should enable efficient processing and analysis on multi-modal data (e.g., video, audio, and text); and (4) On-device AI should be interpretable and reproducible. This project aims to address these challenges by (1) exploring innovative machine learning algorithms (e.g., multi-task learning) for multi-modal data analysis; (2) exploring multi-modal pruning algorithms (reducing the neural network size without compromising accuracy) that can be applied on edge devices; (3) investigating and explaining how pruning works and using the derived theory to guide further pruning optimization; and (4) improving the energy efficiency of on-device AI algorithms and developing energy-aware scheduling algorithms for on-device AI apps.The research outcomes of this project directly benefit mobile users by accelerating the deployment of efficient AI algorithms on edge devices. Texas State University is a large Hispanic Serving Institution (HSI), which provides a unique platform to engage underrepresented students in science, technology, engineering, and math (STEM) research. We will enlist the Texas State University Houston-Louis Stokes Alliance for Minority Participation Scholars Program and the Women in Science and Engineering Program to enhance the research and education experiences of underrepresented students in STEM fields.This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.

深度人工神经网络技术已被广泛用于解决计算机视觉、自然语言处理、语音识别等领域中的许多具有挑战性的任务。如今的深度学习算法大多是为高性能服务器设计的，运行在云中。随着边缘设备(例如移动电话和智能手表)的能力越来越强，以及设备上人工智能(AI)的优势(例如保护隐私、无网络工作、本地实时处理数据)变得更加明显，将AI推向边缘将是不可避免的。然而，边缘设备有限的资源(例如计算、内存和电池)带来了全新的挑战：(1)设备上的人工智能必须在不牺牲精度的情况下保持模型的小尺寸；(2)设备上的人工智能必须保持低功耗；(3)未来的设备上人工智能应该能够对多模式数据(如视频、音频和文本)进行高效的处理和分析；以及(4)设备上的人工智能应该是可解释和可重现的。本项目旨在通过以下方式解决这些挑战：(1)探索用于多模式数据分析的创新机器学习算法(例如，多任务学习)；(2)探索可应用于边缘设备的多模式剪枝算法(在不影响精度的情况下减小神经网络的规模)；(3)调查和解释剪枝的工作原理，并使用所得的理论指导进一步的剪枝优化；(4)提高设备上AI算法的能效，开发面向设备上AI应用的节能调度算法，加速高效AI算法在边缘设备上的部署，使移动用户直接受益。德克萨斯州立大学是一所大型拉美裔服务机构(HSI)，它提供了一个独特的平台，让未被充分代表的学生参与科学、技术、工程和数学(STEM)研究。我们将加入德克萨斯州立大学休斯顿-路易斯·斯托克斯少数群体参与学者联盟计划和女性科学与工程计划，以增强STEM领域未被充分代表的学生的研究和教育经验。该奖项反映了NSF的法定使命，并通过使用基金会的智力优势和更广泛的影响审查标准进行评估，被认为值得支持。

项目成果

Is Pruning Compression?: Investigating Pruning Via Network Layer Similarity

剪枝是压缩吗？：通过网络层相似性研究剪枝

• 发表时间: 2020
• 期刊: IEEE Winter Conference on Applications of Computer Vision (WACV ’20
• 作者: Blakeney, Cody;Yan, Yan;Zong, Ziliang
• 通讯作者: Zong, Ziliang

Craft Distillation: Layer-wise Convolutional Neural Network Distillation

• DOI: 10.1109/cscloud-edgecom49738.2020.00051
• 发表时间: 2020-08
• 期刊: 2020 7th IEEE International Conference on Cyber Security and Cloud Computing (CSCloud)/2020 6th IEEE International Conference on Edge Computing and Scalable Cloud (EdgeCom)
• 作者: Cody Blakeney;Xiaomin Li;Yan Yan-Yan;Ziliang Zong

Learning Omnidirectional Flow in 360° Video via Siamese Representation

通过连体表示学习 360° 视频中的全向流

• 发表时间: 2022
• 期刊: European Conference on Computer Vision
• 作者: Keshav Bhandari;Bin Duan;Gaowen Liu;Hugo Latapie;Ziliang Zong;Yan Yan
• 通讯作者: Yan Yan

Audio-Visual Event Localization via Recursive Fusion by Joint Co-Attention

• DOI: 10.1109/wacv48630.2021.00406
• 发表时间: 2020-08
• 期刊: 2021 IEEE Winter Conference on Applications of Computer Vision (WACV)
• 作者: Bin Duan;Hao Tang;Wei Wang;Ziliang Zong;Guowei Yang;Yan Yan-Yan

Parallel Blockwise Knowledge Distillation for Deep Neural Network Compression

• DOI: 10.1109/tpds.2020.3047003
• 发表时间: 2020-12
• 期刊: IEEE Transactions on Parallel and Distributed Systems
• 影响因子: 5.3
• 作者: Cody Blakeney;Xiaomin Li;Yan Yan-Yan;Ziliang Zong