CIF: Small: Deep Learning for Information Theory- Tackling Algorithm Deficit

信息论为如何以最佳方式交流、压缩和处理信息提供了数学基础。包括通信代码在内的基本算法是利用人类的聪明才智发明的，并通过深入的数学推理证明是最优的。深度学习方法是机器学习的一个分支，它的兴起为重新审视发现新算法的范式提供了机会。该项目研究如何利用深度学习来加速长期存在的信息论问题中的算法发现。在更广泛的范围内，该项目将促进深度学习和信息论之间更强有力的接口，使两个研究界都受益。该项目的成果还将有助于创建一个理论，解释深度学习算法在许多应用领域中的收益，从而解决我们在理解机器学习方面的一个重要科学空白。本项目研究了深度学习和信息论之间的三个问题。(1)基于深度学习的代码设计：这一重点研究代码设计问题，代码被用来处理通信介质中的噪声。这包括首先复制使用这种新范例在代码设计中取得的以前的成功，以及在未解决的问题中发明新的代码--这要求新的网络体系结构可以在代码之外具有应用。(2)基于深度学习的统计特性测试：研究深度学习在信息估计(如互信息估计)和统计特性测试(如独立性测试)中的作用。高维性能测试的解决方案将需要新的p值保证，这将被探索。(3)深度学习的信息论基础：将研究深度学习的基本信息论原则，如样本复杂性和递归神经网络的最佳训练算法。该奖项反映了NSF的法定使命，并通过使用基金会的智力优势和更广泛的影响审查标准进行评估，被认为值得支持。

Information theory provides the mathematical underpinning on how to optimally communicate, compress and process information. The underlying algorithms, including codes for communication, have been invented using human ingenuity and proved to be optimal using deep mathematical reasoning. The rise of deep-learning methods, a branch of machine learning, presents an opportunity to revisit the paradigm for discovering new algorithms. This project studies how to utilize deep-learning for accelerating algorithm discovery in long-standing information-theoretic problems. On a broader scale, this project will promote a stronger interface between deep-learning and information theory, benefiting both research communities. The outcomes of this project will also help create a theory explaining the gains derived by deep-learning algorithms in many application domains, thus addressing an important scientific gap in our understanding of machine-learning.This project studies three problems at the interface of deep-learning and information theory. (1) Deep-Learning based Code Design: This thrust studies the problem of code design, where the code is used to tackle the noise in the communication medium. This involves first replicating the previous successes in code design using this new paradigm, as well as inventing novel codes in unsolved problems -- requiring novel network architectures that can have application beyond codes. (2) Statistical Property Testing with Deep-Learning: How deep-learning can help in information estimation (such as mutual-information estimation) and statistical property testing (such as independence testing) will be studied. Solutions to high-dimensional property testing will require novel p-value guarantees, which will be explored. (3) Information-theoretic Underpinnings of Deep-Learning: The fundamental information theoretic principles underlying deep-learning, such as the sample complexity and optimal training algorithms for recurrent neural networks will be studied.This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.