Collaborative Research: CDS&E-MSS: Deep Network Compression and Continual Learning: Theory and Application

深度神经网络（dnn）已经导致了许多任务的变革性发展，例如识别照片中的人或物体，翻译和合成自然语言，以及生成科学和医学图像。这些进步之所以成为可能，是因为神经网络经过训练，可以在大型数据集中找到模式。平衡深度网络的大小和性能之间的权衡是设计深度神经网络的一个重要方面，它可以很容易地转化为硬件。尽管深度学习在现实问题中产生了显著的性能，但它们消耗了大量的内存和计算资源，这限制了它们的大规模部署。一旦神经网络被训练，它们也会变得脆弱，因为当一个网络在接受新任务的训练时，它通常会忘记以前学过的任务。这种脆弱性给构建人工智能系统带来了挑战，因为人工智能系统的目的是在其整个生命周期中不断学习，这导致花费更多的计算资源来重新训练网络完成它们已经学习过的任务。这导致对电力的大量需求，从而导致碳足迹增加和不利的环境影响。在这个项目中，研究人员提出了新的算法和理论分析，通过压缩神经网络的学习参数并使用这些压缩参数进行持续学习来降低神经网络的功耗。研究生将参与研究并接受跨学科培训。该项目的总体目标是为神经网络压缩开发一种新的概率框架。利用这个框架，研究人员将开发基于过滤器和层之间连接的网络压缩算法，这提供了一个在训练和测试过程中都有效的稀疏化标准。在网络压缩识别出神经网络的哪些参数比其他参数更重要之后，这个特征可以用来开发持续学习的算法，这种算法是有效的，因为重要的参数被优先用于后续任务的学习。研究人员将开发基于单个层的统计数据和层与层之间的连接的持续学习的压缩启发算法。研究者还将以数学定理的形式提供稀疏性分析和理论解释。该奖项反映了美国国家科学基金会的法定使命，并通过使用基金会的知识价值和更广泛的影响审查标准进行评估，被认为值得支持。

Deep neural networks (DNNs) have led to transformative developments in a wide number of tasks, such as identifying people or objects in photographs, translating and synthesizing natural language, and generating scientific and medical images. These advances were possible because neural networks are trained to find patterns in large datasets. Balancing the trade-off between the size and performance of a deep network is a vital aspect of designing deep neural networks that can easily be translated to hardware. Although deep learning yields remarkable performance in real-world problems, they consume a large amount of memory and computational resources, which limits their large-scale deployment. Once neural networks are trained, they can also be brittle in the sense that when a network is trained on a new task, it typically forgets previously learned tasks. This brittleness presents challenges in building artificial intelligence systems that are intended to learn continually throughout their lifespan, and it leads to even more computational resources spent to retrain networks on tasks they have already learned. This results in large demands for electrical power, which leads to an increased carbon footprint and adverse environmental impacts. In this project the investigators propose novel algorithms and theoretical analysis for reducing the power consumption of neural networks by compressing their learned parameters and using these compressed parameters for continual learning. Graduate students will be involved in the research and receive interdisciplinary training.The overall goal of this project is to develop a novel probabilistic framework for neural network compression. Using this framework, the investigators will develop network compression algorithms based on the connectivity between filters and layers, which provides a sparsification criterion that is efficient in both training and testing processes. After network compression identifies which parameters of a neural network are more important than others, this feature can be used to develop algorithms for continual learning which are efficient because the important parameters are prioritized for the learning of subsequent tasks. The investigators will develop compression-inspired algorithms for continual learning based on statistics of individual layers and the connectivity between layers. The investigators will also provide a sparsity analysis and theoretical explanations in the form of mathematical theorems.This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.