Collaborative Research: Intrinsically Motivated Learning in Artificial Agents

协作研究：人工智能中的内在动机学习项目摘要人类有着无尽的好奇心；我们自发地探索和操纵我们的环境，看看我们能让他们做什么；我们从发现和实现中获得乐趣。我们经常为了自己的利益而从事这些活动，而不是作为解决实际问题的步骤。心理学家将这些行为称为内在动机行为，因为奖励是这些活动的内在原因，而不是由于满足更基本的生物需求。但我们在内在动机的行为中所学到的东西，对于我们发展成为有能力的自主实体至关重要，能够在各种实际问题出现时有效地解决它们。该项目的目标是开发一个内在动机学习的计算模型，该模型将允许人工代理构建和扩展胜任自主所需的可重用技能的层次结构。这个项目建立在现有的机器学习研究、大脑奖励系统神经科学的最新进展以及古典和当代动机心理学理论的基础上。该模型的核心是计算强化学习方面的最新理论和算法进展，特别是与技能相关的新概念和用于技能层次学习的新学习算法。该项目开发了一个数学框架，在一系列模拟代理中实现了该模型，并展示了这将在一系列日益复杂的环境中实现的进步。近年来，智能优点机器学习方法变得更加强大。尽管有这些进步和它们的实用性，但今天的学习算法远远达不到机器学习的可能性。它们通常应用于单个、孤立的问题，对于每个问题，它们都必须手动调整，并且必须仔细准备训练数据集。他们没有所需的生成能力来显著扩展他们的能力，使其超越最初的内置表征。它们没有解决许多原因，即学习对于让动物应对新问题如此有用，因为它们会随着时间的延长而出现新的问题。该项目的成功将为机器学习提供一个根本性的进步，并将该领域推向一个新的方向。虽然与内在动机相关的计算研究并不是全新的，但它目前还不发达，并且没有利用计算强化学习领域以及脑奖励和动机系统的神经科学中高度相关的最新进展。此外，计算研究没有利用游戏、好奇心、惊喜和其他涉及内在动机学习的因素的心理学理论。这个项目通过采取一种明确的跨学科方法来解决这些缺点。广泛影响-新方法承诺提高我们以造福社会的方式控制复杂系统行为的能力。机器学习算法在生物信息学、制造、通信、机器人和安全系统等领域的广泛应用中发挥了重要作用。尽快提高机器学习技术的能力在战略上、经济上和智力上都很重要。这个项目试图解决其中的一些挑战。该项目将加强计算机科学的机器学习社区与致力于研究人类认知发展和教育的各学科之间的跨学科联系。拟议研究中关注的具体方法尚未整合。内在动机的心理学研究和机器学习之间几乎没有交叉。拟议的研究将纠正这种情况，从而有助于建立一种能够促进这两个领域未来发展的沟通途径。该项目有可能有助于我们理解人类认知发展的一般原则，并对教育产生影响，其中增强内在动机是一个关键因素。该项目的教育部分侧重于通过培养研究生来进行研究生教育。这包括在马萨诸塞大学和密歇根大学开设跨学科研究生级别的研讨会，由PI教授关于内在动机学习的主题。在招收研究生方面，该项目将利用美国马萨诸塞州大学作为NSF资助的东北联盟的牵头机构所扮演的角色，该联盟支持和指导对科学、数学或工程学科学术生涯感兴趣的少数族裔学生。密歇根大学将特别努力招收本科生，并让他们参与学生项目，这些项目由玛丽安·莎拉·帕克学者项目资助，该项目以女本科生为目标，为夏季研究机会提供资金。

Collaborative Research: Intrinsically Motivated Learning in Artificial AgentsProject SummaryHumans are unendingly curious; we spontaneously explore and manipulate our surroundings to see what we can make them do; we obtain enjoyment from making discoveries and for making things happen. We often engage in these activities for their own sakes rather than as steps toward solving practical problems. Psychologists call these intrinsically motivated behaviors because rewards are intrinsic in these activities instead of being due to the satisfaction of more primary biological needs. But what we learn during intrinsically motivated behavior is essential for our development as competent autonomous entities able to efficiently solve a wide range of practical problems as they arise. This project's objective is to develop a computational model of intrinsically motivated learning that will allow artificial agents to construct and extend hierarchies of reusable skills that are needed for competent autonomy. This project builds on existing research in machine learning, recent advances in the neuroscience of brain reward systems, and classical and contemporary psychological theories of motivation. At the core of the model are recent theoretical and algorithmic advances in computational reinforcement learning, specifically, new concepts related to skills and new learning algorithms for learning with skill hierarchies. The project develops a mathematical framework, implements the model in a series of simulated agents, and demonstrates the advances this will make possible in a series of increasingly complex environments.Intellectual Merit-Machine learning methods have become much more powerful in recent years. Despite these advances and their utility, today's learning algorithms fall far short of the possibilities for machine learning. They are typically applied to single, isolated problems for each of which they have to be hand-tuned and for which training data sets have to be carefully prepared. They do not have the generative capacity required to significantly extend their abilities beyond initially built-in representations. They do not address many of the reasons that learning is so useful in allowing animals to cope with new problems as they arise over extended periods of time. Success in this project will provide a fundamental advance in machine learning and move the field in a new direction. Although computational study related to intrinsic motivation is not entirely new, it is currently underdeveloped and does not take advantage of the highly relevant recent advances in the field of computational reinforcement learning and in the neuroscience of brain reward and motivation systems. Furthermore, computational studies do not take advantage of psychological theories of play, curiosity, surprise, and other factors involved in intrinsically motivated learning. This project addresses these shortcoming by taking an explicitly interdisciplinary approach.Broader Impacts-The new methods promise to improve our ability to control the behavior of complex systems in ways that will benefit society. Machine learning algorithms have been instrumental in a wide variety of applications in such areas as bioinformatics, manufacturing, communications, robotics, and security systems. It is important strategically, economically, and intellectually to increase the power of machine learning technologies as rapidly as possible. This project attempts to address some of these challenges. This project will strengthen interdisciplinary ties between the machine learning community of computer science and various disciplines devoted to the study of human cognitive development and education. The specific methods of concern in the proposed research have not yet been integrated. There has been verylittle cross-fertilization between the psychological study of intrinsic motivation and machine learning. The proposed research will remedy this situation, thereby helping to create an avenue of communication that can foster future developments in both fields. The project has the potential to contribute to our understanding of general principles underlying human cognitive development, with implications for education, where enhancing intrinsic motivation is a key factor.The educational component of the project focuses on graduate education through its training of graduate students. This includes the offering interdisciplinary graduate-level seminars at both U. of Massachusetts and U. of Michigan, to be taught by the PIs on the topic of intrinsically motivated learning. In its recruitment of graduate students, the project will take advantage of the role that U. Massachusetts plays as the lead institution in the NSF funded Northeast Alliance, which supports and mentors underrepresented minority students interested in academic careers in a science, mathematics, or engineering discipline. At U. of Michigan special effort will be made to recruit and involve undergraduates in student projects leading to summer projects funded by the Marian Sarah Parker Scholars Program, which targets female undergraduates and provides funds for summer research opportunities.