Collaborative Research: CDS&E-MSS: Deep Network Compression and Continual Learning: Theory and Application

深度神经网络(DNN)导致了许多任务的变革性发展，如识别照片中的人或物体，翻译和合成自然语言，以及生成科学和医学图像。这些进展之所以成为可能，是因为神经网络经过训练，可以在大数据集中找到模式。在深度神经网络的规模和性能之间进行权衡，是设计易于转化为硬件的深度神经网络的一个重要方面。尽管深度学习在实际问题中表现出色，但它们消耗了大量的内存和计算资源，这限制了它们的大规模部署。一旦训练了神经网络，它们也可能变得脆弱，因为当一个网络就一项新任务进行训练时，它通常会忘记以前学到的任务。这种脆弱性给构建人工智能系统带来了挑战，这些系统旨在在网络的整个生命周期内不断学习，并导致花费更多的计算资源来重新培训网络执行它们已经学习的任务。这导致了对电力的大量需求，从而导致碳足迹增加，并对环境造成不利影响。在这个项目中，研究人员提出了新的算法和理论分析，通过压缩神经网络的学习参数并将压缩后的参数用于连续学习来降低神经网络的功耗。研究生将参与这项研究并接受跨学科的培训。这个项目的总体目标是开发一个新的神经网络压缩的概率框架。使用这个框架，研究人员将开发基于过滤器和层之间的连通性的网络压缩算法，这提供了一个在训练和测试过程中都有效的稀疏化标准。在网络压缩识别出神经网络的哪些参数比其他参数更重要之后，该特征可用于开发用于持续学习的算法，该算法是高效的，因为重要参数被优先用于后续任务的学习。研究人员将根据各个层的统计数据和层之间的连通性，开发受压缩启发的持续学习算法。调查人员还将以数学定理的形式提供稀疏性分析和理论解释。这一裁决反映了NSF的法定使命，并通过使用基金会的智力优势和更广泛的影响审查标准进行评估，被认为值得支持。

Deep neural networks (DNNs) have led to transformative developments in a wide number of tasks, such as identifying people or objects in photographs, translating and synthesizing natural language, and generating scientific and medical images. These advances were possible because neural networks are trained to find patterns in large datasets. Balancing the trade-off between the size and performance of a deep network is a vital aspect of designing deep neural networks that can easily be translated to hardware. Although deep learning yields remarkable performance in real-world problems, they consume a large amount of memory and computational resources, which limits their large-scale deployment. Once neural networks are trained, they can also be brittle in the sense that when a network is trained on a new task, it typically forgets previously learned tasks. This brittleness presents challenges in building artificial intelligence systems that are intended to learn continually throughout their lifespan, and it leads to even more computational resources spent to retrain networks on tasks they have already learned. This results in large demands for electrical power, which leads to an increased carbon footprint and adverse environmental impacts. In this project the investigators propose novel algorithms and theoretical analysis for reducing the power consumption of neural networks by compressing their learned parameters and using these compressed parameters for continual learning. Graduate students will be involved in the research and receive interdisciplinary training.The overall goal of this project is to develop a novel probabilistic framework for neural network compression. Using this framework, the investigators will develop network compression algorithms based on the connectivity between filters and layers, which provides a sparsification criterion that is efficient in both training and testing processes. After network compression identifies which parameters of a neural network are more important than others, this feature can be used to develop algorithms for continual learning which are efficient because the important parameters are prioritized for the learning of subsequent tasks. The investigators will develop compression-inspired algorithms for continual learning based on statistics of individual layers and the connectivity between layers. The investigators will also provide a sparsity analysis and theoretical explanations in the form of mathematical theorems.This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.