CRII: CIF: New Paradigms in Generalization and Information-Theoretic Analysis of Deep Neural Networks

在过去的十年中，深度学习已成为各种机器学习任务的首选方法。数字图书馆的应用领域不断扩大，现在包括自动驾驶汽车、机器人辅助手术、医学成像等许多领域。社会对这类技术的广泛接受有赖于人类理解和信任它们的能力。不幸的是，DL系统非凡的实际有效性并没有与一个全面的理论相结合来解释它们是如何操作的，以及为什么它们在真实世界的数据上如此成功。这种情况阻碍了人工智能在上述应用程序中的更广泛部署。为了缓解这一僵局，该项目试图揭开深度神经网络(DNN)的面纱，使深度神经网络能够实现数据挖掘，并阐明信息在这些系统中是如何处理的。这样做将使人工智能机制的决策对最终用户和其他利益攸关方更加透明，从而有助于他们的理解。通过严格的性能保证，该项目还旨在描述深度学习系统保证不会失败的情况。这些进展将为高性能人工智能系统在我们的日常生活中的集成奠定基础，释放其宝贵的潜在影响。该项目通过一种新的信息论方法解决了数字图书馆理论中的关键挑战。主要目的是阐明DNN逐步构建表示的过程-从浅层的粗略和过度冗余的表示，到较深层的高度聚集和可解释的表示-并让设计者对该过程有更多的控制。为此，进行了三个协同推进。首先是通过量化通过DNN的信息流来开发内部表示的新的复杂性度量。至关重要的是，这些措施旨在对计算机视觉、语音和文本处理的最先进网络中典型的层维度进行有效计算。第二个重点是通过新的依赖于实例的泛化界限，将所开发的复杂性度量与网络的泛化能力联系起来。这里的目标是在可有效计算的品质因数方面为给定的DNN提供性能保证。最后，进一步利用开发的机器来构建工具，用于剪枝冗余神经元/层，可视化DNN的操作，并提高DNN的可解释性。总之，这项研究致力于将当前DNN设计的不确定试错过程推进到确定性工程实践领域。该奖项反映了NSF的法定使命，并通过使用基金会的智力优势和更广泛的影响审查标准进行评估，被认为值得支持。

Over the past decade, deep learning (DL) has become the method of choice for various machine learning tasks. The realm of DL applications constantly expands, now including autonomous vehicles, robotic-assisted surgery, medical imaging, and many others. A wide societal acceptance of such technologies relies on the ability of humans to understand and trust them. Unfortunately, the exceptional practical effectiveness of DL systems is not coupled with a comprehensive theory to explain how they operate and why they are so successful on real-world data. This state of affairs obstructs a wider deployment of AI for the applications described above. To alleviate this impasse, this project seeks to open the hood of Deep Neural Networks (DNNs) that enable DL and elucidate how information is processed in these systems. Doing so would make the decisions of AI mechanisms more transparent to end users and other stakeholders, thus contributing to their understanding. Via rigorous performance guarantees, this project also aims to characterize the circumstances under which deep learning system are warranted not to fail. These advances will set the stage for the integration of high-performance AI systems in our daily lives, unlocking their invaluable potential impact. The project tackles key challenges in DL theory via a novel information-theoretic approach. The main objective is to shed light on the process by which DNNs progressively build representations --- from crude and over-redundant representations in shallow layers, to highly-clustered and interpretable ones in deeper layers --- and to give the designer more control over that process. To that end, three synergistic thrusts are pursued. First is developing novel complexity measures of internal representations by quantifying the flow of information through the DNN. Crucially, these measures are designed for efficient computation over layer dimensionalities typical to state-of-the-art networks for computer vision, speech, and text processing. The second thrust focuses on relating the developed complexity measures to the generalization capability of the network via new instance-dependent generalization bounds. The goal here is to provide performance guarantees for a given DNN in terms of efficiently computable figures of merit. Lastly, the developed machinery is further leveraged to construct tools for pruning redundant neurons/layers, visualizing the DNN's operation, and progressing DNN interpretability. Altogether, this research strives to progress the current uncertain trial-and-error process of DNN design towards the domain of deterministic engineering practice.This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.