CRCNS: Representational foundations of adaptive behavior in natural and artificial

 描述(申请人提供)：概述：计算神经科学中最著名的成功案例之一是发现决策的许多方面可以根据强化学习(RL)的形式框架来理解。来自RL的想法揭示了学习和动作选择中的许多行为现象，揭示了奖励驱动行为背后的功能解剖学和神经过程，以及神经调节功能的基本方面。然而，尽管取得了所有这些成功，基于RL的工作却受到一个令人不快的事实的困扰：标准的RL算法不能很好地扩展到大型、复杂的问题。如果人类的学习和决策是由类似RL的机制驱动的，那么我们如何处理日常生活中典型的丰富、大规模的任务？心理学和神经科学的现有研究都暗示了这个问题的一个答案：如果决策者对任务有紧凑的、智能的格式表示，复杂的问题就可以被克服。这一原理体现在对国际象棋专家游戏的研究中，这些研究表明，国际象棋大师利用棋盘结构的高度整合的内部表征；在对额叶和顶叶功能的研究中，揭示了任务偶发事件强烈塑造的接受区；以及对海马体的研究，表明这种结构在支持任务空间的分层组织的“认知图谱”中所起的作用。 并非巧合的是，随着人们对降维、层次和深度学习技术的日益感兴趣，表示在基于RL的机器学习和机器人学研究中的关键作用日益突出。 本项目旨在系统地、经验性地解释表征在支持RL和目标导向行为中的作用。该项目汇集了三名在认知和计算神经科学(Botvinick，Gershman)以及机器学习和机器人学(Konidaris)方面具有互补专业知识的研究人员。我们共同提出了一个综合的、跨学科的研究计划，应用人类受试者的行为和神经成像工作，神经生理和行为数据的计算建模，正式的数学工作和使用人工代理进行模拟。拟议的研究在主题和方法上是不同的，但共同努力形成一个既有形式上的基础，又有经验限制的理论。在更具体的层面上，我们的研究集中在四类特定的表示上，考虑了每一类表示对RL的计算影响，以及每一类表示与神经科学和人类行为的相关性。正如我们在项目描述中所详细描述的，这些方法包括(1)度量嵌入、(2)谱分解、(3)层次表示和(4)符号表示。除了分别研究这四种表示形式的含义外，我们还假设它们合在一起 一种分层系统，作为一个整体来支持有时相互冲突的学习和动作控制需求。 智力优势(由申请者提供)：理解表征结构如何影响学习和决策是认知科学、行为神经科学和人工智能领域的核心挑战。如果在这一领域建立一个计算明确、经验主义验证的理论，并特别关注表征在R中的作用，这将是一项具有广泛影响的重要成就。正如我们之前的研究所表明的那样，利用机器学习的概念工具来研究人类行为和大脑功能的策略可以提供相当大的科学杠杆。拟议的工作受到既定研究工作的推动，并以此为基础，将这些研究工作结合在一起，以利用协同增效的机会。除了回答具体的实证和计算问题外，拟议的工作还旨在为未来在一个重要调查领域的研究开辟新的途径。 更广泛的影响(由申请者提供)：拟议的工作位于神经科学、心理学、人工智能和机器学习的十字路口，并有望促进这些领域之间日益增长的交流。该项目将不同学科背景的调查人员聚集在一起，明确目标是在知识分子之间架起一座桥梁。鉴于其与认知和发展心理学、行为、认知和系统神经科学以及行为经济学的相关性，拟议中的工作可能会找到广泛的科学受众。然而，这项工作可能会在人工智能、机器学习和机器人学中引起同等的兴趣，目前的挑战恰恰是理解表示学习如何允许RL扩大到大型问题。RL的代表性方法已经在行业内引起了强烈的兴趣，目前的调查人员有积极参与的记录。拟议工作的主题也适用于其他领域，包括教育和培训，以及军事和医疗决策支持。该项目的计划在研究生和博士后层面都有一个强有力的培训部分，致力于促进代表人数不足的少数群体的参与以及国际参与。

 DESCRIPTION (provided by applicant): Overview: Among the most celebrated success stories in computational neuroscience is the discovery that many aspects of decision-making can be understood in terms of the formal framework of reinforcement learning (RL). Ideas drawn from RL have shed light on many behavioral phenomena in learning and action selection, on the functional anatomy and neural processes underlying reward-driven behavior, and on fundamental aspects of neuromodulatory function. However, for all these successes, RL-based work is haunted by an inconvenient truth: Standard RL algorithms scale poorly to large, complex problems. If human learning and decision-making are driven by RL-like mechanisms, how is it that we cope with the kinds of rich, large-scale tasks that are typical of everyday life? Existing research in both psychology and neuroscience hints at one answer to this question: Complex problems can be conquered if the decision-maker is equipped with compact, intelligently formatted representations of the task. This principle is seen in studies of expert play in chess, which show that chess masters leverage highly integrative internal representations of board configurations; in studies of frontal and parietal lobe function, which have revealed receptive fields strongly shaped by task contingencies; and studies on the hippocampus, which point to the role of this structure in supporting a hierarchically organized 'cognitive map,' of task space. Not coincidentally, the critical role of representation has come increasingly to the fore in RL-based research in machine learning and robotics, with growing interest in techniques for dimensionality reduction, hierarchy and deep learning. The present project aims toward a systematic, empirically validated account of the role of representation in supporting RL and goal-directed behavior at large. The project brings together three investigators with complementary expertise in cognitive and computational neuroscience (Botvinick, Gershman) and machine learning and robotics (Konidaris). Together, we propose an integrative, interdisciplinary program of research, applying behavioral and neuroimaging work with human subjects, computational modeling of neurophysiological and behavioral data, formal mathematical work and simulations with artificial agents. The proposed studies are diverse in theme and method, but work together toward a theory that is both formally grounded and empirically constrained. At a more concrete level, our research focuses on four specific classes of representation, considering the computational impact of each for RL, as well as the relevance of each to neuroscience and human behavior. As detailed in our Project Description, these include (1) metric embedding, (2) spectral decomposition, (3) hierarchical representation and (4) symbolic representation. In addition to investigating the implications of each of these four forms of representation individually, we hypothesize that they fit together into a tiered system, which works as a whole to support the sometimes competing demands of learning and action control. Intellectual Merit (provided by applicant): Understanding how representational structure impacts learning and decision making is a core challenge in cognitive science, behavioral neuroscience and, artificial intelligence. Success in establishing a computationally explicit, empirically validated theory in this area, with a specific focus on the role of representation in R, would represent an important achievement with wide repercussions. The strategy of leveraging conceptual tools from machine learning to investigate human behavior and brain function can offer considerable scientific leverage, as our own previous research illustrates. The proposed work is motivated by and builds upon established lines of research, bringing these together in order to capitalize on opportunities for synergy. In addition to answering specific empirical and computational questions, the proposed work aims to open up new avenues for future research in an important area of inquiry. Broader Impact (provided by applicant): The proposed work lies at the crossroads of neuroscience, psychology, artificial intelligence and machine learning, and promises to advance the growing exchange among these fields. The project brings together investigators with contrasting disciplinary affiliations, with the explicit goal of bridging between intellectual cultres. The proposed work is likely to find a wide scientific audience, given its relevance to cognitive and developmental psychology, behavioral, cognitive and systems neuroscience, and behavioral economics. However, the work is likely to be of equal interest within artificial intelligence, machine learning, and robotics, where a current challenge is precisely to understand how representation learning can allow RL to scale up to large problems. Representational approaches to RL are already of intense interest within industry, where the present investigators have a record of active engagement. The topic of the proposed work has applicability in other areas as well, including education and training, and military and medical decision support. The plan for the project has a robust training component at both graduate and postdoctoral levels, with a commitment to fostering involvement of underrepresented minorities, as well as international engagement.