Toward Machine Competence: Combining Demonstration-based and Experience-based Machine Learning

数字数据的不断扩展与前所未有的计算能力相结合，创造了新的机会，通过对大量数据收集的分析以及与现有系统或人员的广泛互动，推动计算机解释(如自然语言处理和计算机感知)和计算机决策。尽管机器学习的进步导致了人工智能最近的进步，但目前的机器学习方法在根本上仍然有限：它们要么基于模仿明确的人类演示，要么依赖于从强化中自我发现-每一种方法本身都是不够的。人类通过指导、模仿和经验的结合来获得能力，但机器学习方法通常是在这些角度之间孤立的。 这一研究计划将解决开发能够通过基于经验的学习和基于演示的学习相结合来获得能力的算法的挑战。为了使有效的一体化建立在坚实的基础上，这项研究还将解决在每个支持分领域中出现的核心问题；特别是从示范中学习(例如监督)和从经验中学习(例如加强)。主要集中在：(1)统一基于价值和策略的强化学习；(2)正向和反向强化学习的关联；(3)扩展基于策略和非策略强化学习方法以开发用于结构化产出预测的演示和评估预言；(4)利用博弈论中的均衡概念。 特别是，对于(1)我最近发展了一种基于值的强化学习和基于策略的强化学习的新统一，基于一种观察，即当存在熵正则化时，动作值和策略概率是二元的。这一统一还提出了有效的新方法来结合正向和反向强化学习，以及加速学习的策略上和策略外的数据，这构成了(2)的基础。这些研究的一个关键方面将是更有效地利用演示，这本质上是非策略的，并将所产生的技术应用于自然语言处理、组合优化和程序综合中自然出现的结构化输出预测问题，从而实现(3)。最后，对于(4)，我将利用我最近开发的深度学习和博弈论之间的新联系，这些联系允许改进深度学习和强化学习方法的稳定性和稀疏性。

The relentless expansion of digital data combined with unprecedented computing power has created new opportunities to advance computer interpretation (e.g. natural language processing and computer perception) and computer decision making, through the analysis of massive data collections and extensive interaction with existing systems or people. Even though advances in machine learning have led to the recent progress in artificial intelligence, current machine learning methods remain limited in a fundamental way: they are either based on mimicking explicit human demonstration or rely on self discovery from reinforcement---each of which is inadequate on its own. Humans achieve competence through a combination of instruction, imitation and experience, yet machine learning methods are typically siloed between these perspectives. This research program will address the challenge of developing algorithms that can acquire competence through the integration of experience-based and demonstration-based learning. To base an effective integration on sound foundations, this research will also address core questions that arise in each of the supporting subareas; in particular, learning from demonstration (e.g. supervision) and learning from experience (e.g. reinforcement). The main foci are (1) unifying value and policy based reinforcement learning, (2) relating forward and inverse reinforcement learning, (3) extending on-policy and off-policy reinforcement learning methods to exploit demonstrations and evaluation oracles for structured output prediction, and (4) exploiting equilibrium concepts from game theory. In particular, for (1) I have recently developed a new unification of value-based and policy-based reinforcement learning, based on an observation that action values and policy probabilities are duals when entropy regularization is present. This unification also suggests effective new methods for combining forward and inverse reinforcement learning, and on-policy and off-policy data to accelerate learning, which form the basis for (2). A key aspect of these investigations will be to make more effective use of demonstrations, which are inherently off-policy, and apply the resulting techniques to structured output prediction problems that arise naturally in natural language processing, combinatorial optimization, and program synthesis, fulfilling (3). Finally, for (4) I will exploit novel connections between deep learning and game theory I have recently developed, which allow for improved stability and sparsity in deep and reinforcement learning methods.