Toward Machine Competence: Combining Demonstration-based and Experience-based Machine Learning

点击翻译按钮获取中文摘要

The relentless expansion of digital data combined with unprecedented computing power has created new opportunities to advance computer interpretation (e.g. natural language processing and computer perception) and computer decision making, through the analysis of massive data collections and extensive interaction with existing systems or people. Even though advances in machine learning have led to the recent progress in artificial intelligence, current machine learning methods remain limited in a fundamental way: they are either based on mimicking explicit human demonstration or rely on self discovery from reinforcement---each of which is inadequate on its own. Humans achieve competence through a combination of instruction, imitation and experience, yet machine learning methods are typically siloed between these perspectives.******This research program will address the challenge of developing algorithms that can acquire competence through the integration of experience-based and demonstration-based learning. To base an effective integration on sound foundations, this research will also address core questions that arise in each of the supporting subareas; in particular, learning from demonstration (e.g. supervision) and learning from experience (e.g. reinforcement). The main foci are (1) unifying value and policy based reinforcement learning, (2) relating forward and inverse reinforcement learning, (3) extending on-policy and off-policy reinforcement learning methods to exploit demonstrations and evaluation oracles for structured output prediction, and (4) exploiting equilibrium concepts from game theory.******In particular, for (1) I have recently developed a new unification of value-based and policy-based reinforcement learning, based on an observation that action values and policy probabilities are duals when entropy regularization is present. This unification also suggests effective new methods for combining forward and inverse reinforcement learning, and on-policy and off-policy data to accelerate learning, which form the basis for (2). A key aspect of these investigations will be to make more effective use of demonstrations, which are inherently off-policy, and apply the resulting techniques to structured output prediction problems that arise naturally in natural language processing, combinatorial optimization, and program synthesis, fulfilling (3). Finally, for (4) I will exploit novel connections between deep learning and game theory I have recently developed, which allow for improved stability and sparsity in deep and reinforcement learning methods.