CIF: Small: Deep Learning for Information Theory- Tackling Algorithm Deficit

CIF：小型：信息论深度学习 - 解决算法缺陷

• 批准号: 1908003
• 负责人: Sreeram Kannan
• 金额: $ 45.84万
• 依托单位: University of Washington
• 依托单位国家: 美国
• 项目类别: Standard Grant
• 财政年份: 2019
• 资助国家: 美国
• 起止时间: 2019-07-01 至 2022-06-30
• 项目状态: 已结题
• 来源: https://www.nsf.gov/awardsearch/showAward?AWD_ID=1908003&HistoricalAwards=false
• 关键词: CIF; Small; Deep; Learning; Information

Information theory provides the mathematical underpinning on how to optimally communicate, compress and process information. The underlying algorithms, including codes for communication, have been invented using human ingenuity and proved to be optimal using deep mathematical reasoning. The rise of deep-learning methods, a branch of machine learning, presents an opportunity to revisit the paradigm for discovering new algorithms. This project studies how to utilize deep-learning for accelerating algorithm discovery in long-standing information-theoretic problems. On a broader scale, this project will promote a stronger interface between deep-learning and information theory, benefiting both research communities. The outcomes of this project will also help create a theory explaining the gains derived by deep-learning algorithms in many application domains, thus addressing an important scientific gap in our understanding of machine-learning.This project studies three problems at the interface of deep-learning and information theory. (1) Deep-Learning based Code Design: This thrust studies the problem of code design, where the code is used to tackle the noise in the communication medium. This involves first replicating the previous successes in code design using this new paradigm, as well as inventing novel codes in unsolved problems -- requiring novel network architectures that can have application beyond codes. (2) Statistical Property Testing with Deep-Learning: How deep-learning can help in information estimation (such as mutual-information estimation) and statistical property testing (such as independence testing) will be studied. Solutions to high-dimensional property testing will require novel p-value guarantees, which will be explored. (3) Information-theoretic Underpinnings of Deep-Learning: The fundamental information theoretic principles underlying deep-learning, such as the sample complexity and optimal training algorithms for recurrent neural networks will be studied.This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.

信息论为如何最佳地交流、压缩和处理信息提供了数学基础。底层的算法，包括通信代码，都是利用人类的聪明才智发明的，并通过深入的数学推理证明是最佳的。深度学习方法是机器学习的一个分支，它的兴起为重新审视发现新算法的范式提供了机会。该项目研究如何利用深度学习来加速长期存在的信息理论问题中的算法发现。在更广泛的范围内，该项目将促进深度学习和信息理论之间更强大的接口，使两个研究社区受益。该项目的成果还将有助于创建一个理论来解释深度学习算法在许多应用领域中所获得的收益，从而填补我们对机器学习理解的一个重要科学空白。该项目研究深度学习和信息论接口的三个问题。(1)基于深度学习的代码设计：这一方向研究代码设计问题，其中代码用于解决通信介质中的噪声。这涉及到首先使用这种新的范式复制以前在代码设计中的成功，以及在未解决的问题中发明新的代码-需要新的网络架构，可以超越代码的应用。(2)深度学习的统计属性测试：深度学习如何帮助信息估计（如互信息估计）和统计属性测试（如独立性测试）。高维属性测试的解决方案将需要新的p值保证，这将被探索。(3)深度学习的信息论基础：研究深度学习的基本信息论原理，如样本复杂性和递归神经网络的最佳训练算法。该奖项反映了NSF的法定使命，并通过使用基金会的知识价值和更广泛的影响审查标准进行评估，被认为值得支持。

项目成果

Deepcode: Feedback Codes via Deep Learning

• DOI: 10.1109/jsait.2020.2986752
• 发表时间: 2018-07
• 期刊: IEEE Journal on Selected Areas in Information Theory
• 作者: Hyeji Kim;Yihan Jiang;Sreeram Kannan;Sewoong Oh;P. Viswanath

Turbo Autoencoder: Deep learning based channel codes for point-to-point communication channels

• 发表时间: 2019-11
• 期刊: ArXiv
• 作者: Yihan Jiang;Hyeji Kim;Himanshu Asnani;Sreeram Kannan;Sewoong Oh;P. Viswanath

Learning in Gated Neural Networks

• 发表时间: 2019-06
• 作者: Ashok Vardhan Makkuva;Sewoong Oh;Sreeram Kannan;P. Viswanath

C-MI-GAN : Estimation of Conditional Mutual Information using MinMax formulation

• 发表时间: 2020-05
• 期刊: ArXiv
• 作者: A. Mondal;A. Bhattacharya;Sudipto Mukherjee;Sreeram Kannan;Himanshu Asnani;A. Prathosh