CIF: Small: Information Theory Meets Deep Learning: Universal Probability and Common Information for High-Dimensional Data

信息论是一门科学学科，它研究我们如何能够很好地沟通，压缩和处理各种信号和数据。在过去的70年里，信息理论的原则推动了我们日常生活的数字化转型，通过无线网络进行无处不在的可靠的0和1通信是各种公共和私营企业不可或缺的操作。然而，除了这些“比特”之外，信息论中的传统方法在处理多维、大字母和复杂空间依赖的真实世界数据时经常遇到一些限制。该项目旨在开发新的数学工具和工程技术，通过将深度学习纳入信息论的原则来处理如此复杂的现实世界数据。在这种协同组合中，信息理论提供了性能保证和复杂问题的系统分解。反过来，深度学习为高效处理高维数据提供了工作程序。该项目探索了两个具体的研究方向，其中结合提出了新的数据科学框架。第一个方向建立在普适概率的概念上，这是数据未知分布的密切代理。尽管基于普适概率的传统算法对于高维、大字母表、长距离相关数据收敛得相当慢，但所提出的框架利用深度神经网络，通过平滑地聚合它们来有效地学习多个数据上下文的普适概率分布。由此产生的信息理论和深度学习的结合提供了一种新的范式，以原则性但务实的方式解决一般数据科学问题，并具有理论性能保证和实际性能改进。第二个方向受到信息论中公共信息概念的启发，并开发了一种新的神经网络模型，用于多个高维数据集之间的统计推断（例如，自动绘制一张最好地描述给定文本的图片）。该模型和相关的训练算法可应用于各种数据处理任务，如联合和条件生成以及高维输出数据的监督或半监督学习。该奖项反映了NSF的法定使命，并通过使用基金会的知识价值和更广泛的影响审查标准进行评估，被认为值得支持。

Information theory is a scientific discipline that studies how well we can communicate, compress, and otherwise process various signals and data. Over the past 70 years, the tenets of information theory have driven the digital transformation of our daily lives, with ubiquitous and reliable communication of 0s and 1s over wireless networks as an indispensable operation in diverse public and private enterprises. Beyond such "bits," however, conventional methods in information theory often encounter several limitations when dealing with real-world data of multiple dimensions, large alphabets, and complex spatial dependence. This project aims to develop new mathematical tools and engineering techniques for handling such complicated real-world data by incorporating deep learning into the tenets of information theory. In this synergistic combination, information theory provides performance guarantees and a systematic decomposition of complex problems. Deep learning, in turn, provides working procedures for efficient processing of high-dimensional data.This project explores two concrete research directions in which the combination prompts new data science frameworks. The first direction builds on the notion of universal probability, which is a close proxy to the unknown distribution of the data. Although conventional algorithms based on universal probability converge rather slowly for high-dimensional, large alphabet, long-range dependent data, the proposed framework leverages deep neural networks to efficiently learn the universal probability distribution across multiple data contexts by aggregating them smoothly. The resulting combination of information theory and deep learning provides a new paradigm to address general data science problems in a principled yet pragmatic manner, with theoretical performance guarantees and practical performance improvements. The second direction is inspired by the notion of common information in information theory, and develops a new neural network model for statistical inference among multiple high-dimensional data sets (for example, drawing a picture automatically that describes a given text the best). This model and associated training algorithms can be applied to various data processing tasks such as joint and conditional generation as well as supervised or semisupervised learning of high-dimensional output data.This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.