Information-theoretic analysis and synthesis in deep learning

点击翻译按钮获取中文摘要

The methodology of deep neural networks, or deep learning, has revolutionized the field of machine learning and now prevails in numerous fields of applied sciences. Despite its great successes, the working mechanism of deep learning is still poorly understood to date. As a consequence, practical applications of this methodology primarily rely on intuitions and heuristics.  In this project, we aim at developing new theoretical understanding of deep learning and inventing novel, more effective, and principled techniques for modelling with deep neural networks and training of such models.  A key hypothesis of this project is that information theory, a mathematical theory originally developed for the study of data compression and data transmission, can be used as an effective tool for the analysis and synthesis in deep learning.  Built on the concepts of entropy and mutual information, which characterize notions of information, information theory possesses a powerful set of analytic techniques that have allowed the determination of the fundamental limits of data compression and data communication. In the deep learning era, information-theoretic concepts and techniques have also been applied to the construction and training of neural network models. More recently information theory has also been demonstrated as a promising tool to theorize deep learning and analyze learning algorithms.  In this project, we propose to use information theory, jointly with other mathematical techniques and computer simulations, to study deep learning. Specifically, this research aims to achieve the following objectives. 1) Developing better understanding of the generalization behaviour of deep neural networks. 2) Developing more effective network architectures and training methods for deep learning. 3) Developing principled and more effective data-dependent regularization schemes. This project will provide new scientific knowledge to the research in deep learning. It will result in innovative and more effective learning techniques and offer fundamental insights and important guidelines to applying deep learning in practice.  The success of this project will further strengthen Canada's leading position in AI research. The techniques developed in this research may benefit Canadian industry in developing innovative AI technologies.