Deep Learning of Representations

点击翻译按钮获取中文摘要

Artificial intelligence requires computers to have knowledge of the world around us, knowledge they can use to answer questions and take decisions. Knowledge can be either hard-wired or learned from data and usually both sources of information are used. Machine learning algorithms endow computers with the ability to acquire knowledge from examples. In the age of big data, there is a great potential in the ability to tap this immense source of information. Representation learning algorithms are machine learning algorithms which also involve the learning of representations. The data (such as an image or a document) seen by a computer must be represented, and better representations make it easier to capture the statistical dependencies present and to learn to answer questions. Whereas representations are traditionally crafted by hand and engineered, recent years have shown that they can be learned and this can have drastic impact on the predictive ability of the learners. Deep learning, or learning of deep representations, is the subject of this proposal. It involves learning multiple levels of representation, corresponding to different levels of abstractions. In the past five years there has been a tremendous impact of research on deep learning, both at an academic level and with industrial breakthroughs. This proposal asks what are the current qualitative deficiencies of the current deep learning algorithms and proposes ideas for how to move beyond these limitations and towards AI. The main limitations that are discussed here regard the ways to scale up, because current models still have the size and capabilities of insect brains and do not even reach the perceptual abilities of rodents. Although faster and larger computers will matter, it is hypothesized that such advances in hardware will not suffice. For example, parallelization of the current learning procedures is not trivial and yet this is where computing power continues to grow. There also seem to be numerical optimization challenges that cannot be solved by simply pouring ten times more compute power, such as real or apparent local minima arising in the learning dynamics, for which it is not sufficient to restart thousands of times the optimization procedures: the result remains poor unless a different initialization procedure is used. There are also fundamental challenges regarding unsupervised learning: whereas most empirical progress of recent years has been with supervised learning (where humans have labeled data and told the computer what to answer to many questions), the greatest promise for future breakthroughs may come from unsupervised learning procedures (coupled with supervised learning). The main challenge there arises out of the intractability of the normalization constant involved in probabilistic models with many random variables and many high-probability modes separated by vast regions of low probability, a situation that is the most common in artificial intelligence applications. Although Monte-Carlo Markov chain methods are the most general solutions to this problem, new and maybe radically different ways of addressing the problem may be required and will be explored. Finally, this proposal considers the very basic question of what is a good representation, and proposes to view the different strategies we have to obtain good representations as priors that can help the learner in the very important task of disentangling the underlying factors of variation, i.e., of figuring out and separating from each other the factors that explain the data. As in past research of the applicant, these algorithmic explorations will be evaluated on challenging applications involving real data, from images and video to natural language text, or combining multiple modalities.