CRII: CIF: New Paradigms in Generalization and Information-Theoretic Analysis of Deep Neural Networks

在过去的十年中，深度学习（DL）已经成为各种机器学习任务的首选方法。深度学习的应用领域不断扩大，现在包括自动驾驶汽车、机器人辅助手术、医学成像等。社会对这些技术的广泛接受依赖于人类理解和信任它们的能力。不幸的是，深度学习系统卓越的实际有效性并没有与一个全面的理论相结合，来解释它们是如何运作的，以及为什么它们在现实世界的数据上如此成功。这种情况阻碍了上述应用程序更广泛地部署AI。为了缓解这一僵局，该项目试图打开深度神经网络（dnn）的面纱，使深度学习成为可能，并阐明这些系统中信息是如何处理的。这样做将使人工智能机制的决策对最终用户和其他利益相关者更加透明，从而有助于他们的理解。通过严格的性能保证，该项目还旨在描述深度学习系统保证不会失败的情况。这些进步将为高性能人工智能系统融入我们的日常生活奠定基础，释放其宝贵的潜在影响。该项目通过一种新颖的信息理论方法解决了深度学习理论中的关键挑战。主要目标是阐明dnn逐步构建表征的过程——从浅层的粗糙和过度冗余的表征，到更深层的高度聚集和可解释的表征——并给予设计师更多的控制这一过程。为此目的，采取了三项协同努力。首先是通过量化通过深度神经网络的信息流来开发新的内部表征的复杂性度量。至关重要的是，这些措施的设计是为了在最先进的计算机视觉、语音和文本处理网络中典型的层维上进行有效计算。第二个重点是通过新的实例依赖的泛化界限将开发的复杂性度量与网络的泛化能力联系起来。这里的目标是为给定的深度神经网络提供有效可计算的性能保证。最后，开发的机器进一步利用构建工具来修剪冗余神经元/层，可视化DNN的操作，并提高DNN的可解释性。总之，本研究力求将当前深度神经网络设计的不确定试错过程推进到确定性工程实践领域。该奖项反映了美国国家科学基金会的法定使命，并通过使用基金会的知识价值和更广泛的影响审查标准进行评估，被认为值得支持。

Over the past decade, deep learning (DL) has become the method of choice for various machine learning tasks. The realm of DL applications constantly expands, now including autonomous vehicles, robotic-assisted surgery, medical imaging, and many others. A wide societal acceptance of such technologies relies on the ability of humans to understand and trust them. Unfortunately, the exceptional practical effectiveness of DL systems is not coupled with a comprehensive theory to explain how they operate and why they are so successful on real-world data. This state of affairs obstructs a wider deployment of AI for the applications described above. To alleviate this impasse, this project seeks to open the hood of Deep Neural Networks (DNNs) that enable DL and elucidate how information is processed in these systems. Doing so would make the decisions of AI mechanisms more transparent to end users and other stakeholders, thus contributing to their understanding. Via rigorous performance guarantees, this project also aims to characterize the circumstances under which deep learning system are warranted not to fail. These advances will set the stage for the integration of high-performance AI systems in our daily lives, unlocking their invaluable potential impact. The project tackles key challenges in DL theory via a novel information-theoretic approach. The main objective is to shed light on the process by which DNNs progressively build representations --- from crude and over-redundant representations in shallow layers, to highly-clustered and interpretable ones in deeper layers --- and to give the designer more control over that process. To that end, three synergistic thrusts are pursued. First is developing novel complexity measures of internal representations by quantifying the flow of information through the DNN. Crucially, these measures are designed for efficient computation over layer dimensionalities typical to state-of-the-art networks for computer vision, speech, and text processing. The second thrust focuses on relating the developed complexity measures to the generalization capability of the network via new instance-dependent generalization bounds. The goal here is to provide performance guarantees for a given DNN in terms of efficiently computable figures of merit. Lastly, the developed machinery is further leveraged to construct tools for pruning redundant neurons/layers, visualizing the DNN's operation, and progressing DNN interpretability. Altogether, this research strives to progress the current uncertain trial-and-error process of DNN design towards the domain of deterministic engineering practice.This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.