Information-theoretic analysis and synthesis in deep learning

深度学习中的信息论分析与综合

• 批准号: RGPIN-2020-06285
• 负责人: Mao, Yongyi
• 金额: $ 2.4万
• 依托单位: University of Ottawa
• 依托单位国家: 加拿大
• 项目类别: Discovery Grants Program - Individual
• 财政年份: 2021
• 资助国家: 加拿大
• 起止时间: 2021-01-01 至 2022-12-31
• 项目状态: 已结题
• 来源: https://www.nserc-crsng.gc.ca/ase-oro/Details-Detailles_eng.asp?id=741139
• 关键词: Information; theoretic; analysis; synthesis; deep

The methodology of deep neural networks, or deep learning, has revolutionized the field of machine learning and now prevails in numerous fields of applied sciences. Despite its great successes, the working mechanism of deep learning is still poorly understood to date. As a consequence, practical applications of this methodology primarily rely on intuitions and heuristics.  In this project, we aim at developing new theoretical understanding of deep learning and inventing novel, more effective, and principled techniques for modelling with deep neural networks and training of such models.  A key hypothesis of this project is that information theory, a mathematical theory originally developed for the study of data compression and data transmission, can be used as an effective tool for the analysis and synthesis in deep learning.  Built on the concepts of entropy and mutual information, which characterize notions of information, information theory possesses a powerful set of analytic techniques that have allowed the determination of the fundamental limits of data compression and data communication. In the deep learning era, information-theoretic concepts and techniques have also been applied to the construction and training of neural network models. More recently information theory has also been demonstrated as a promising tool to theorize deep learning and analyze learning algorithms.  In this project, we propose to use information theory, jointly with other mathematical techniques and computer simulations, to study deep learning. Specifically, this research aims to achieve the following objectives. 1) Developing better understanding of the generalization behaviour of deep neural networks. 2) Developing more effective network architectures and training methods for deep learning. 3) Developing principled and more effective data-dependent regularization schemes. This project will provide new scientific knowledge to the research in deep learning. It will result in innovative and more effective learning techniques and offer fundamental insights and important guidelines to applying deep learning in practice.  The success of this project will further strengthen Canada's leading position in AI research. The techniques developed in this research may benefit Canadian industry in developing innovative AI technologies.

深度神经网络或深度学习的方法已经彻底改变了机器学习领域，现在在许多应用科学领域都很流行。尽管取得了巨大的成功，但迄今为止，人们对深度学习的工作机制仍然知之甚少。因此，这种方法的实际应用主要依赖于直觉和直觉。在这个项目中，我们的目标是发展对深度学习的新的理论理解，并发明新的，更有效的，原则性的技术，用于使用深度神经网络建模和训练这样的模型。这个项目的一个关键假设是，信息论，一种最初为研究数据压缩和数据传输而开发的数学理论，可用作深度学习中分析和综合的有效工具。基于熵和互信息的概念，它们表征了信息的概念，信息理论拥有一套强大的分析技术，可以确定数据压缩和数据通信的基本限制。在深度学习时代，信息理论的概念和技术也被应用于神经网络模型的构建和训练。最近，信息论也被证明是一种有前途的工具，以理论化的深度学习和分析学习算法。在这个项目中，我们建议使用信息论，结合其他数学技术和计算机模拟，研究深度学习。具体而言，本研究旨在实现以下目标。1)更好地理解深度神经网络的泛化行为。2)为深度学习开发更有效的网络架构和训练方法。3)开发原则性的和更有效的数据依赖正则化方案。该项目将为深度学习的研究提供新的科学知识。它将带来创新和更有效的学习技术，并为在实践中应用深度学习提供基本见解和重要指导。该项目的成功将进一步加强加拿大在人工智能研究方面的领先地位。这项研究开发的技术可能有利于加拿大工业开发创新的人工智能技术。