Deep Learning of Representations

人工智能要求计算机了解我们周围的世界，这些知识可以用来回答问题和做出决定。知识既可以是硬连线的，也可以是从数据中学习的，通常这两种信息来源都被使用。机器学习算法赋予计算机从示例中获取知识的能力。在大数据时代，挖掘这一巨大信息来源的能力具有巨大潜力。表示学习算法是也涉及表示学习的机器学习算法。计算机看到的数据（如图像或文档）必须被表示，更好的表示可以更容易地捕获存在的统计依赖关系并学习回答问题。虽然表征传统上是手工制作和设计的，但近年来已经表明它们是可以学习的，这可能对学习者的预测能力产生巨大影响。深度学习或深度表征的学习是这个提议的主题。它涉及到学习多层次的表示，对应于不同层次的抽象。在过去的五年里，深度学习的研究产生了巨大的影响，无论是在学术层面还是在工业上都取得了突破。该提案提出了当前深度学习算法的质量缺陷，并提出了如何超越这些限制并走向人工智能的想法。这里讨论的主要限制是关于扩大规模的方法，因为目前的模型仍然具有昆虫大脑的大小和能力，甚至没有达到啮齿动物的感知能力。虽然更快更大的计算机将是重要的，但据推测，硬件的这种进步是不够的。例如，当前学习过程的并行化并不是微不足道的，但这正是计算能力持续增长的地方。还有一些数值优化的挑战是无法通过简单地投入十倍的计算能力来解决的，例如在学习动态中出现的真实的或明显的局部最小值，对于这些挑战，重新启动数千次优化过程是不够的：除非使用不同的初始化过程，否则结果仍然很差。无监督学习也面临着根本性的挑战：尽管近年来大多数经验性的进展都是在监督学习（人类标记数据并告诉计算机如何回答许多问题）方面，但未来突破的最大希望可能来自无监督学习程序（加上监督学习）。主要的挑战来自于概率模型中涉及的归一化常数的棘手性，该模型具有许多随机变量和许多高概率模式，这些模式被大量的低概率区域分隔开，这种情况在人工智能应用中最常见。虽然蒙特-卡罗马尔可夫链方法是这个问题的最一般的解决方案，新的，也许根本不同的方式来解决这个问题可能需要，并将探讨。最后，这个建议考虑了什么是好的表征这个非常基本的问题，并建议将我们必须获得好的表征的不同策略视为先验，这些策略可以帮助学习者完成解开变化的潜在因素这一非常重要的任务，即，找出并将解释数据的因素相互分离。与申请人过去的研究一样，这些算法探索将在涉及从图像和视频到自然语言文本的真实的数据或组合多种模态的挑战性应用上进行评估。

Artificial intelligence requires computers to have knowledge of the world around us, knowledge they can use to answer questions and take decisions. Knowledge can be either hard-wired or learned from data and usually both sources of information are used. Machine learning algorithms endow computers with the ability to acquire knowledge from examples. In the age of big data, there is a great potential in the ability to tap this immense source of information. Representation learning algorithms are machine learning algorithms which also involve the learning of representations. The data (such as an image or a document) seen by a computer must be represented, and better representations make it easier to capture the statistical dependencies present and to learn to answer questions. Whereas representations are traditionally crafted by hand and engineered, recent years have shown that they can be learned and this can have drastic impact on the predictive ability of the learners. Deep learning, or learning of deep representations, is the subject of this proposal. It involves learning multiple levels of representation, corresponding to different levels of abstractions. In the past five years there has been a tremendous impact of research on deep learning, both at an academic level and with industrial breakthroughs. This proposal asks what are the current qualitative deficiencies of the current deep learning algorithms and proposes ideas for how to move beyond these limitations and towards AI. The main limitations that are discussed here regard the ways to scale up, because current models still have the size and capabilities of insect brains and do not even reach the perceptual abilities of rodents. Although faster and larger computers will matter, it is hypothesized that such advances in hardware will not suffice. For example, parallelization of the current learning procedures is not trivial and yet this is where computing power continues to grow. There also seem to be numerical optimization challenges that cannot be solved by simply pouring ten times more compute power, such as real or apparent local minima arising in the learning dynamics, for which it is not sufficient to restart thousands of times the optimization procedures: the result remains poor unless a different initialization procedure is used. There are also fundamental challenges regarding unsupervised learning: whereas most empirical progress of recent years has been with supervised learning (where humans have labeled data and told the computer what to answer to many questions), the greatest promise for future breakthroughs may come from unsupervised learning procedures (coupled with supervised learning). The main challenge there arises out of the intractability of the normalization constant involved in probabilistic models with many random variables and many high-probability modes separated by vast regions of low probability, a situation that is the most common in artificial intelligence applications. Although Monte-Carlo Markov chain methods are the most general solutions to this problem, new and maybe radically different ways of addressing the problem may be required and will be explored. Finally, this proposal considers the very basic question of what is a good representation, and proposes to view the different strategies we have to obtain good representations as priors that can help the learner in the very important task of disentangling the underlying factors of variation, i.e., of figuring out and separating from each other the factors that explain the data. As in past research of the applicant, these algorithmic explorations will be evaluated on challenging applications involving real data, from images and video to natural language text, or combining multiple modalities.