Deep Learning of Representations

人工智能要求计算机了解我们周围的世界，这些知识可以用来回答问题和做出决定。知识既可以是硬连接的，也可以是从数据中学习的，通常两种信息来源都使用。机器学习算法赋予计算机从示例中获取知识的能力。在大数据时代，挖掘这一巨大信息来源的能力具有巨大的潜力。表示学习算法是机器学习算法，也涉及表示学习。计算机看到的数据(如图像或文档)必须被表示，而更好的表示使捕获存在的统计相关性和学习回答问题变得更容易。虽然传统上的表征是手工制作和设计的，但最近几年的研究表明，它们是可以学习的，这可能会对学习者的预测能力产生巨大影响。深度学习，或深度表征的学习，是这项提议的主题。它涉及到学习多个级别的表示，对应于不同级别的抽象。在过去的五年里，深度学习的研究产生了巨大的影响，无论是在学术层面还是在行业突破方面。这份提案询问了目前深度学习算法的质量缺陷是什么，并提出了如何超越这些限制，走向人工智能的想法。这里讨论的主要限制是关于扩大规模的方法，因为目前的模型仍然具有昆虫大脑的大小和能力，甚至无法达到啮齿动物的感知能力。尽管更快、更大的计算机将是重要的，但据推测，硬件方面的这些进步是不够的。例如，当前学习过程的并行化并不是微不足道的，但这是计算能力继续增长的地方。似乎还有一些数值优化挑战不能通过简单地倾注十倍的计算能力来解决，例如学习动态中出现的真实或明显的局部最小值，对于这些挑战，重新启动数千次优化过程是不够的：除非使用不同的初始化过程，否则结果仍然很差。关于无监督学习也存在根本性的挑战：尽管近年来的大多数经验进展都是在监督学习(人类给数据贴上标签，并告诉计算机如何回答许多问题)方面取得的，但未来突破的最大希望可能来自无监督学习过程(再加上监督学习)。这里的主要挑战来自于概率模型中涉及的归一化常数的难解性，该模型包含许多随机变量和许多高概率模式，由大量的低概率区域分隔，这种情况在人工智能应用中最常见。尽管蒙特卡洛马尔可夫链方法是解决这一问题的最普遍的方法，但可能需要并将探索解决该问题的新的、也许是根本不同的方法。最后，这个建议考虑了什么是好的表征这一非常基本的问题，并建议将我们获得好的表征所必须的不同策略视为先验，这些策略可以帮助学习者在非常重要的任务中解开变化的潜在因素，即找出并彼此分离解释数据的因素。与申请人过去的研究一样，这些算法探索将在涉及真实数据的挑战性应用程序上进行评估，从图像和视频到自然语言文本，或组合多种模式。

Artificial intelligence requires computers to have knowledge of the world around us, knowledge they can use to answer questions and take decisions. Knowledge can be either hard-wired or learned from data and usually both sources of information are used. Machine learning algorithms endow computers with the ability to acquire knowledge from examples. In the age of big data, there is a great potential in the ability to tap this immense source of information. Representation learning algorithms are machine learning algorithms which also involve the learning of representations. The data (such as an image or a document) seen by a computer must be represented, and better representations make it easier to capture the statistical dependencies present and to learn to answer questions. Whereas representations are traditionally crafted by hand and engineered, recent years have shown that they can be learned and this can have drastic impact on the predictive ability of the learners. Deep learning, or learning of deep representations, is the subject of this proposal. It involves learning multiple levels of representation, corresponding to different levels of abstractions. In the past five years there has been a tremendous impact of research on deep learning, both at an academic level and with industrial breakthroughs. This proposal asks what are the current qualitative deficiencies of the current deep learning algorithms and proposes ideas for how to move beyond these limitations and towards AI. The main limitations that are discussed here regard the ways to scale up, because current models still have the size and capabilities of insect brains and do not even reach the perceptual abilities of rodents. Although faster and larger computers will matter, it is hypothesized that such advances in hardware will not suffice. For example, parallelization of the current learning procedures is not trivial and yet this is where computing power continues to grow. There also seem to be numerical optimization challenges that cannot be solved by simply pouring ten times more compute power, such as real or apparent local minima arising in the learning dynamics, for which it is not sufficient to restart thousands of times the optimization procedures: the result remains poor unless a different initialization procedure is used. There are also fundamental challenges regarding unsupervised learning: whereas most empirical progress of recent years has been with supervised learning (where humans have labeled data and told the computer what to answer to many questions), the greatest promise for future breakthroughs may come from unsupervised learning procedures (coupled with supervised learning). The main challenge there arises out of the intractability of the normalization constant involved in probabilistic models with many random variables and many high-probability modes separated by vast regions of low probability, a situation that is the most common in artificial intelligence applications. Although Monte-Carlo Markov chain methods are the most general solutions to this problem, new and maybe radically different ways of addressing the problem may be required and will be explored. Finally, this proposal considers the very basic question of what is a good representation, and proposes to view the different strategies we have to obtain good representations as priors that can help the learner in the very important task of disentangling the underlying factors of variation, i.e., of figuring out and separating from each other the factors that explain the data. As in past research of the applicant, these algorithmic explorations will be evaluated on challenging applications involving real data, from images and video to natural language text, or combining multiple modalities.