CRII: RI: RUI: Principled Methods for Compressing Neural Networks through Discrete Optimization and Polyhedral Theory

机器学习的最新进展通过利用目前由现代计算机产生、存储和分析的海量数据，产生了更准确的基于软件的预测。其中许多进展都归功于深度学习，通过深度学习，可用的数据被用来训练人工神经网络，其中每个输入都由多层人工神经元顺序分析，以识别更复杂的关系。当预测任务更具挑战性或预测需要更准确时，需要使用通用图形处理器(GPU)等专用硬件来训练相当大的网络。然而，资源更有限的个人和组织可能无法访问GPU，并且这些较大的网络可能不适合物联网(IoT)和移动设备等嵌入式系统。一方面，有许多不精确的剪枝方法来减小训练后的神经网络的规模。这些方法可能会降低准确性，在应用于稍有修改的数据时影响网络的稳健性，并导致公平性问题，因为修剪的效果是不均匀的，并且不成比例地影响数据中代表不足的组。另一方面，训练有素的神经网络所代表的关系往往并不像它们可能表现的那样复杂。这意味着有可能获得代表相同关系的更小的神经网络，从而避免传统修剪方法的副作用。这个项目旨在提高我们对神经网络可以代表什么，以及如何准确地压缩它们以更有效地使用的理解。该项目旨在开发精确的神经网络压缩算法，并利用多面体理论和离散优化技术研究网络对线性区域数量的表达能力与网络可压缩性之间的关系。我们的主要目标是开发更快和更可扩展的算法来识别对训练的神经网络表示的模型具有有限影响或没有影响的网络修改。其次，我们的目标是确定可表现性和可压缩性之间的理论联系，并开发更有效的方法来测量线性区域的数量。该奖项反映了NSF的法定使命，并通过使用基金会的智力优势和更广泛的影响审查标准进行评估，被认为值得支持。

Recent advances in machine learning have led to more accurate software-based predictions by leveraging the vast amounts of data that are currently produced, stored, and analyzed by modern computers. Many of these advances are due to deep learning, through which the available data is used to train artificial neural networks in which every input is sequentially analyzed by many layers of artificial neurons to identify more complex relationships. When the prediction task is more challenging or the predictions need to be more accurate, considerably larger networks are trained with dedicated hardware such as general-purpose Graphics Processing Units (GPUs). Nevertheless, individuals and organizations with more constrained resources may not have access to GPUs, and those larger networks may not fit in embedded systems such as Internet of Things (IoT) and mobile devices. On the one hand, there are many inexact pruning methods for reducing the size of a neural network after training. These methods may reduce the accuracy, affect the robustness of the network when applied to slightly modified data, and lead to fairness issues because the effect of pruning is uneven and disproportionally affects groups that are underrepresented in the data. On the other hand, the relationships that trained neural networks represent are often not as complex as they could potentially be. That implies that it is possible to obtain smaller neural networks representing the same relationships, hence avoiding the side effects of conventional pruning methods. This project aims to improve our understanding of what neural networks can represent, and how they can be exactly compressed for a more efficient use. This project aims to develop exact neural network compression algorithms and investigate the relationship between network expressiveness in terms of the number of linear regions and network compressibility by leveraging polyhedral theory and discrete optimization techniques. Our primary goal is to develop faster and more scalable algorithms to identify network modifications having limited or no effect to the model represented by trained neural networks. Secondarily, we aim to identify theoretical connections between representability and compressibility as well as develop more efficient methods for measuring the number of linear regions.This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.