Collaborative Research: CDS&E-MSS: Deep Network Compression and Continual Learning: Theory and Application

合作研究：CDS

• 批准号: 2053448
• 负责人: Paul Hand
• 金额: $ 15万
• 依托单位: Northeastern University
• 依托单位国家: 美国
• 项目类别: Continuing Grant
• 财政年份: 2021
• 资助国家: 美国
• 起止时间: 2021-07-01 至 2024-06-30
• 项目状态: 已结题
• 来源: https://www.nsf.gov/awardsearch/showAward?AWD_ID=2053448&HistoricalAwards=false
• 关键词: Collaborative; Research; CDS&E; MSS; Deep

Deep neural networks (DNNs) have led to transformative developments in a wide number of tasks, such as identifying people or objects in photographs, translating and synthesizing natural language, and generating scientific and medical images. These advances were possible because neural networks are trained to find patterns in large datasets. Balancing the trade-off between the size and performance of a deep network is a vital aspect of designing deep neural networks that can easily be translated to hardware. Although deep learning yields remarkable performance in real-world problems, they consume a large amount of memory and computational resources, which limits their large-scale deployment. Once neural networks are trained, they can also be brittle in the sense that when a network is trained on a new task, it typically forgets previously learned tasks. This brittleness presents challenges in building artificial intelligence systems that are intended to learn continually throughout their lifespan, and it leads to even more computational resources spent to retrain networks on tasks they have already learned. This results in large demands for electrical power, which leads to an increased carbon footprint and adverse environmental impacts. In this project the investigators propose novel algorithms and theoretical analysis for reducing the power consumption of neural networks by compressing their learned parameters and using these compressed parameters for continual learning. Graduate students will be involved in the research and receive interdisciplinary training.The overall goal of this project is to develop a novel probabilistic framework for neural network compression. Using this framework, the investigators will develop network compression algorithms based on the connectivity between filters and layers, which provides a sparsification criterion that is efficient in both training and testing processes. After network compression identifies which parameters of a neural network are more important than others, this feature can be used to develop algorithms for continual learning which are efficient because the important parameters are prioritized for the learning of subsequent tasks. The investigators will develop compression-inspired algorithms for continual learning based on statistics of individual layers and the connectivity between layers. The investigators will also provide a sparsity analysis and theoretical explanations in the form of mathematical theorems.This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.

深度神经网络（DNN）导致了许多任务的变革性发展，例如在照片中识别人或对象，翻译和综合自然语言，并产生科学和医学图像。 这些进步之所以可能，是因为对神经网络进行了训练以在大型数据集中找到模式。平衡深层网络的规模和性能之间的权衡是设计深神经网络的重要方面，这些神经网络可以轻松地转化为硬件。尽管深度学习在现实世界中产生了出色的表现，但它们会消耗大量的内存和计算资源，从而限制了其大规模部署。一旦训练了神经网络，它们也可能会变得脆弱，因为当网络接受新任务培训时，它通常会忘记以前学习的任务。 这种脆弱性在构建旨在在整个生命周期中不断学习的人工智能系统方面面临着挑战，这导致了花费更多的计算资源来重新研究他们已经学到的任务的网络。 这导致对电力的需求很大，从而导致碳足迹增加和不利的环境影响。在该项目中，研究人员提出了新颖的算法和理论分析，以通过压缩他们学习的参数并使用这些压缩参数来持续学习来降低神经网络的功耗。 研究生将参与研究并接受跨学科培训。该项目的总体目标是为神经网络压缩开发一种新颖的概率框架。 使用此框架，研究人员将基于过滤器和层之间的连通性开发网络压缩算法，该连通性提供了在培训和测试过程中有效的稀疏标准。 在网络压缩确定了神经网络的哪些参数比其他参数更为重要之后，该功能可用于开发持续学习的算法，这是有效的，因为重要参数已优先考虑以学习后续任务。研究人员将根据单个层的统计数据和层之间的连通性开发以压缩启发的算法来进行连续学习。 调查人员还将以数学定理的形式提供稀疏分析和理论解释。该奖项反映了NSF的法定任务，并使用基金会的知识分子优点和更广泛的影响审查标准，被认为值得通过评估来提供支持。