Deep Learning of Representations

人工智能需要计算机了解我们周围的世界，这些知识可以用来回答问题和做出决定。知识可以是天生的，也可以是从数据中学习的，通常这两种信息来源都会用到。机器学习算法赋予计算机从实例中获取知识的能力。在大数据时代，挖掘这一巨大信息源的潜力巨大。表征学习算法是机器学习算法，它也涉及表征的学习。计算机看到的数据（如图像或文档）必须被表示出来，更好的表示可以更容易地捕获存在的统计依赖性并学习回答问题。虽然表征传统上是手工制作和设计的，但近年来已经表明它们是可以学习的，这可能对学习者的预测能力产生重大影响。深度学习，或深度表征的学习，是这个提议的主题。它涉及到学习多个层次的表示，对应于不同层次的抽象。在过去的五年里，深度学习的研究产生了巨大的影响，无论是在学术层面还是在工业上都取得了突破。本提案询问当前深度学习算法的质量缺陷是什么，并提出了如何超越这些限制并走向人工智能的想法。这里讨论的主要限制是关于扩大规模的方法，因为目前的模型仍然具有昆虫大脑的大小和能力，甚至没有达到啮齿动物的感知能力。尽管更快、更大的计算机很重要，但据推测，硬件上的这些进步还不够。例如，当前学习过程的并行化不是微不足道的，但这是计算能力持续增长的地方。似乎还有一些数值优化挑战不能通过简单地投入十倍的计算能力来解决，例如学习动力学中出现的实际或明显的局部最小值，对于这些问题，重新启动数千次优化过程是不够的：除非使用不同的初始化过程，否则结果仍然很差。关于无监督学习也存在根本性的挑战：尽管近年来大多数的经验进展都是在监督学习（人类标记数据并告诉计算机如何回答许多问题），但未来最大的突破可能来自无监督学习过程（加上监督学习）。这里的主要挑战来自于包含许多随机变量和许多由大量低概率区域分隔的高概率模式的概率模型中涉及的归一化常数的棘手性，这种情况在人工智能应用中最常见。虽然蒙特卡洛马尔可夫链方法是解决这个问题的最通用的方法，但可能需要新的、可能完全不同的方法来解决这个问题，并将被探索。最后，本建议考虑了什么是好的表征这一非常基本的问题，并建议将我们获得好的表征的不同策略视为先验，这些先验可以帮助学习者完成非常重要的任务，即解开变异的潜在因素，即找出并分离解释数据的因素。与申请人过去的研究一样，这些算法探索将在涉及真实数据的具有挑战性的应用中进行评估，从图像和视频到自然语言文本，或者结合多种模式。

Artificial intelligence requires computers to have knowledge of the world around us, knowledge they can use to answer questions and take decisions. Knowledge can be either hard-wired or learned from data and usually both sources of information are used. Machine learning algorithms endow computers with the ability to acquire knowledge from examples. In the age of big data, there is a great potential in the ability to tap this immense source of information. Representation learning algorithms are machine learning algorithms which also involve the learning of representations. The data (such as an image or a document) seen by a computer must be represented, and better representations make it easier to capture the statistical dependencies present and to learn to answer questions. Whereas representations are traditionally crafted by hand and engineered, recent years have shown that they can be learned and this can have drastic impact on the predictive ability of the learners. Deep learning, or learning of deep representations, is the subject of this proposal. It involves learning multiple levels of representation, corresponding to different levels of abstractions. In the past five years there has been a tremendous impact of research on deep learning, both at an academic level and with industrial breakthroughs. This proposal asks what are the current qualitative deficiencies of the current deep learning algorithms and proposes ideas for how to move beyond these limitations and towards AI. The main limitations that are discussed here regard the ways to scale up, because current models still have the size and capabilities of insect brains and do not even reach the perceptual abilities of rodents. Although faster and larger computers will matter, it is hypothesized that such advances in hardware will not suffice. For example, parallelization of the current learning procedures is not trivial and yet this is where computing power continues to grow. There also seem to be numerical optimization challenges that cannot be solved by simply pouring ten times more compute power, such as real or apparent local minima arising in the learning dynamics, for which it is not sufficient to restart thousands of times the optimization procedures: the result remains poor unless a different initialization procedure is used. There are also fundamental challenges regarding unsupervised learning: whereas most empirical progress of recent years has been with supervised learning (where humans have labeled data and told the computer what to answer to many questions), the greatest promise for future breakthroughs may come from unsupervised learning procedures (coupled with supervised learning). The main challenge there arises out of the intractability of the normalization constant involved in probabilistic models with many random variables and many high-probability modes separated by vast regions of low probability, a situation that is the most common in artificial intelligence applications. Although Monte-Carlo Markov chain methods are the most general solutions to this problem, new and maybe radically different ways of addressing the problem may be required and will be explored. Finally, this proposal considers the very basic question of what is a good representation, and proposes to view the different strategies we have to obtain good representations as priors that can help the learner in the very important task of disentangling the underlying factors of variation, i.e., of figuring out and separating from each other the factors that explain the data. As in past research of the applicant, these algorithmic explorations will be evaluated on challenging applications involving real data, from images and video to natural language text, or combining multiple modalities.