CRII: RI: RUI: Principled Methods for Compressing Neural Networks through Discrete Optimization and Polyhedral Theory

机器学习的最新进展通过利用现代计算机当前产生、存储和分析的大量数据，实现了更准确的基于软件的预测。这些进步中有许多是由于深度学习，通过深度学习，可用数据被用于训练人工神经网络，其中每个输入都被多层人工神经元顺序分析，以识别更复杂的关系。当预测任务更具挑战性或预测需要更准确时，使用专用硬件（如通用图形处理单元（GPU））训练相当大的网络。然而，具有更多受限资源的个人和组织可能无法访问GPU，并且那些较大的网络可能不适合嵌入式系统，例如物联网（IoT）和移动的设备。一方面，有许多不精确的修剪方法用于在训练后减小神经网络的大小。这些方法可能会降低准确性，在应用于稍微修改的数据时影响网络的鲁棒性，并导致公平性问题，因为修剪的效果是不均匀的，并且会影响数据中代表性不足的群体。另一方面，经过训练的神经网络所代表的关系通常并不像它们可能的那样复杂。这意味着可以获得表示相同关系的较小神经网络，从而避免传统修剪方法的副作用。该项目旨在提高我们对神经网络可以表示什么的理解，以及如何准确压缩它们以更有效地使用。该项目旨在开发精确的神经网络压缩算法，并利用多面体理论和离散优化技术研究线性区域数量和网络可压缩性之间的关系。我们的主要目标是开发更快、更可扩展的算法，以识别对训练神经网络所代表的模型影响有限或没有影响的网络修改。其次，我们的目标是确定可表示性和可压缩性之间的理论联系，以及开发更有效的方法来测量线性区域的数量。该奖项反映了NSF的法定使命，并已被认为是值得通过使用基金会的智力价值和更广泛的影响审查标准进行评估的支持。

Recent advances in machine learning have led to more accurate software-based predictions by leveraging the vast amounts of data that are currently produced, stored, and analyzed by modern computers. Many of these advances are due to deep learning, through which the available data is used to train artificial neural networks in which every input is sequentially analyzed by many layers of artificial neurons to identify more complex relationships. When the prediction task is more challenging or the predictions need to be more accurate, considerably larger networks are trained with dedicated hardware such as general-purpose Graphics Processing Units (GPUs). Nevertheless, individuals and organizations with more constrained resources may not have access to GPUs, and those larger networks may not fit in embedded systems such as Internet of Things (IoT) and mobile devices. On the one hand, there are many inexact pruning methods for reducing the size of a neural network after training. These methods may reduce the accuracy, affect the robustness of the network when applied to slightly modified data, and lead to fairness issues because the effect of pruning is uneven and disproportionally affects groups that are underrepresented in the data. On the other hand, the relationships that trained neural networks represent are often not as complex as they could potentially be. That implies that it is possible to obtain smaller neural networks representing the same relationships, hence avoiding the side effects of conventional pruning methods. This project aims to improve our understanding of what neural networks can represent, and how they can be exactly compressed for a more efficient use. This project aims to develop exact neural network compression algorithms and investigate the relationship between network expressiveness in terms of the number of linear regions and network compressibility by leveraging polyhedral theory and discrete optimization techniques. Our primary goal is to develop faster and more scalable algorithms to identify network modifications having limited or no effect to the model represented by trained neural networks. Secondarily, we aim to identify theoretical connections between representability and compressibility as well as develop more efficient methods for measuring the number of linear regions.This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.