Information-theoretic analysis and synthesis in deep learning

深度神经网络或深度学习的方法论已经彻底改变了机器学习领域，现在盛行于应用科学的许多领域。尽管深度学习取得了巨大的成功，但人们对深度学习的工作机制仍然知之甚少。因此，这种方法的实际应用主要依赖于直觉和启发式。在这个项目中，我们的目标是发展对深度学习的新的理论理解，并发明用深度神经网络建模和训练这种模型的新颖、更有效和有原则的技术。该项目的一个关键假设是，信息论--最初为研究数据压缩和数据传输而开发的数学理论--可以作为深度学习中分析和综合的有效工具。它建立在表征信息概念的熵和互信息的概念上，信息论拥有一套强大的分析技术，可以确定数据压缩和数据通信的基本界限。在深度学习时代，信息论的概念和技术也被应用于神经网络模型的构建和训练。最近，信息论也被证明是一种很有前途的工具来理论深度学习和分析学习算法。在这个项目中，我们建议使用信息论，结合其他数学技术和计算机模拟，来研究深度学习。具体地说，本研究旨在实现以下目标。1)更好地理解深度神经网络的泛化行为。2)为深度学习开发更有效的网络体系结构和训练方法。3)开发原则性和更有效的数据依赖正则化方案。该项目将为深度学习的研究提供新的科学知识。它将带来创新和更有效的学习技术，并为将深度学习应用于实践提供基本见解和重要指导。该项目的成功将进一步巩固加拿大在人工智能研究方面的领先地位。这项研究开发的技术可能有助于加拿大工业开发创新的人工智能技术。

The methodology of deep neural networks, or deep learning, has revolutionized the field of machine learning and now prevails in numerous fields of applied sciences. Despite its great successes, the working mechanism of deep learning is still poorly understood to date. As a consequence, practical applications of this methodology primarily rely on intuitions and heuristics.  In this project, we aim at developing new theoretical understanding of deep learning and inventing novel, more effective, and principled techniques for modelling with deep neural networks and training of such models.  A key hypothesis of this project is that information theory, a mathematical theory originally developed for the study of data compression and data transmission, can be used as an effective tool for the analysis and synthesis in deep learning.  Built on the concepts of entropy and mutual information, which characterize notions of information, information theory possesses a powerful set of analytic techniques that have allowed the determination of the fundamental limits of data compression and data communication. In the deep learning era, information-theoretic concepts and techniques have also been applied to the construction and training of neural network models. More recently information theory has also been demonstrated as a promising tool to theorize deep learning and analyze learning algorithms.  In this project, we propose to use information theory, jointly with other mathematical techniques and computer simulations, to study deep learning. Specifically, this research aims to achieve the following objectives. 1) Developing better understanding of the generalization behaviour of deep neural networks. 2) Developing more effective network architectures and training methods for deep learning. 3) Developing principled and more effective data-dependent regularization schemes. This project will provide new scientific knowledge to the research in deep learning. It will result in innovative and more effective learning techniques and offer fundamental insights and important guidelines to applying deep learning in practice.  The success of this project will further strengthen Canada's leading position in AI research. The techniques developed in this research may benefit Canadian industry in developing innovative AI technologies.