CRCNS: Representational foundations of adaptive behavior in natural and artificial

CRCNS：自然和人工适应性行为的代表性基础

• 批准号: 9052441
• 负责人: Nathaniel Douglass Daw
• 金额: $ 42.55万
• 依托单位: PRINCETON UNIVERSITY
• 依托单位国家: 美国
• 财政年份: 2015
• 资助国家: 美国
• 起止时间: 2015-09-01 至 2018-05-31
• 项目状态: 已结题
• 来源: https://reporter.nih.gov/project-details/9052441
• 关键词: Accounting; Achievement; Adaptive Behaviors; Algorithms; Anatomy; Area; Artificial Intelligence; Behavior; Behavioral; Brain; Cognitive; Cognitive Science; Complex; Computer Simulation; Data; Decision Making; Fostering; Foundations; Goals; Hippocampus (Brain); Human; Industry; International; Learning; Life; Light; Machine Learning; Maps; Medical; Methods; Military Personnel; Neurosciences; Parietal Lobe; Play; Process; Psychological reinforcement; Psychology; Research; Research Personnel; Rewards; Robotics; Role; Shapes; Structure; System; Techniques; Training; Training and Education; Underrepresented Minority; Work; base; behavioral economics; cognitive neuroscience; cognitive system; computational neuroscience; coping; developmental psychology; driving behavior; frontal lobe; human subject; interest; neuroimaging; neurophysiology; neuroregulation; programs; receptive field; relating to nervous system; scale up; simulation; success; theories; tool

 DESCRIPTION (provided by applicant): Overview: Among the most celebrated success stories in computational neuroscience is the discovery that many aspects of decision-making can be understood in terms of the formal framework of reinforcement learning (RL). Ideas drawn from RL have shed light on many behavioral phenomena in learning and action selection, on the functional anatomy and neural processes underlying reward-driven behavior, and on fundamental aspects of neuromodulatory function. However, for all these successes, RL-based work is haunted by an inconvenient truth: Standard RL algorithms scale poorly to large, complex problems. If human learning and decision-making are driven by RL-like mechanisms, how is it that we cope with the kinds of rich, large-scale tasks that are typical of everyday life? Existing research in both psychology and neuroscience hints at one answer to this question: Complex problems can be conquered if the decision-maker is equipped with compact, intelligently formatted representations of the task. This principle is seen in studies of expert play in chess, which show that chess masters leverage highly integrative internal representations of board configurations; in studies of frontal and parietal lobe function, which have revealed receptive fields strongly shaped by task contingencies; and studies on the hippocampus, which point to the role of this structure in supporting a hierarchically organized 'cognitive map,' of task space. Not coincidentally, the critical role of representation has come increasingly to the fore in RL-based research in machine learning and robotics, with growing interest in techniques for dimensionality reduction, hierarchy and deep learning. The present project aims toward a systematic, empirically validated account of the role of representation in supporting RL and goal-directed behavior at large. The project brings together three investigators with complementary expertise in cognitive and computational neuroscience (Botvinick, Gershman) and machine learning and robotics (Konidaris). Together, we propose an integrative, interdisciplinary program of research, applying behavioral and neuroimaging work with human subjects, computational modeling of neurophysiological and behavioral data, formal mathematical work and simulations with artificial agents. The proposed studies are diverse in theme and method, but work together toward a theory that is both formally grounded and empirically constrained. At a more concrete level, our research focuses on four specific classes of representation, considering the computational impact of each for RL, as well as the relevance of each to neuroscience and human behavior. As detailed in our Project Description, these include (1) metric embedding, (2) spectral decomposition, (3) hierarchical representation and (4) symbolic representation. In addition to investigating the implications of each of these four forms of representation individually, we hypothesize that they fit together into a tiered system, which works as a whole to support the sometimes competing demands of learning and action control. Intellectual Merit (provided by applicant): Understanding how representational structure impacts learning and decision making is a core challenge in cognitive science, behavioral neuroscience and, artificial intelligence. Success in establishing a computationally explicit, empirically validated theory in this area, with a specific focus on the role of representation in R, would represent an important achievement with wide repercussions. The strategy of leveraging conceptual tools from machine learning to investigate human behavior and brain function can offer considerable scientific leverage, as our own previous research illustrates. The proposed work is motivated by and builds upon established lines of research, bringing these together in order to capitalize on opportunities for synergy. In addition to answering specific empirical and computational questions, the proposed work aims to open up new avenues for future research in an important area of inquiry. Broader Impact (provided by applicant): The proposed work lies at the crossroads of neuroscience, psychology, artificial intelligence and machine learning, and promises to advance the growing exchange among these fields. The project brings together investigators with contrasting disciplinary affiliations, with the explicit goal of bridging between intellectual cultres. The proposed work is likely to find a wide scientific audience, given its relevance to cognitive and developmental psychology, behavioral, cognitive and systems neuroscience, and behavioral economics. However, the work is likely to be of equal interest within artificial intelligence, machine learning, and robotics, where a current challenge is precisely to understand how representation learning can allow RL to scale up to large problems. Representational approaches to RL are already of intense interest within industry, where the present investigators have a record of active engagement. The topic of the proposed work has applicability in other areas as well, including education and training, and military and medical decision support. The plan for the project has a robust training component at both graduate and postdoctoral levels, with a commitment to fostering involvement of underrepresented minorities, as well as international engagement.

 描述（由申请人提供）：概述：在计算神经科学中最著名的成功故事之一是发现决策的许多方面可以根据强化学习（RL）的正式框架来理解。从强化学习中得出的想法揭示了学习和动作选择中的许多行为现象，奖励驱动行为背后的功能解剖学和神经过程，以及神经调节功能的基本方面。然而，对于所有这些成功，基于RL的工作被一个令人不安的事实所困扰：标准RL算法对大型复杂问题的扩展性很差。如果人类的学习和决策是由类似RL的机制驱动的，那么我们是如何科普日常生活中典型的丰富的大规模任务的呢？心理学和神经科学的现有研究暗示了这个问题的一个答案：如果决策者配备了紧凑的，智能格式化的任务表示，复杂的问题可以被征服。这一原则在对国际象棋专家的研究中得到了体现，这些研究表明，国际象棋大师利用了棋盘结构的高度整合的内部表征;在对额叶和顶叶功能的研究中，这些研究揭示了感受野受到任务偶然性的强烈影响;在对海马体的研究中，这些研究指出了海马体结构在支持分层组织的任务空间“认知地图”中的作用。 并非巧合的是，表示的关键作用在机器学习和机器人技术中基于RL的研究中越来越突出，对降维，层次结构和深度学习技术的兴趣越来越大。 本项目旨在对表征在支持强化学习和目标导向行为中的作用进行系统的、经验验证的解释。该项目汇集了三名在认知和计算神经科学（Botvinick，Gershman）以及机器学习和机器人技术（Konidaris）方面具有互补专业知识的研究人员。我们共同提出了一个综合的跨学科研究计划，将行为和神经成像工作应用于人类受试者，神经生理和行为数据的计算建模，正式的数学工作和人工代理的模拟。拟议的研究在主题和方法上是多样的，但共同努力形成一个既有形式基础又有经验约束的理论。在更具体的层面上，我们的研究集中在四个特定的表示类别上，考虑每个类别对RL的计算影响，以及每个类别与神经科学和人类行为的相关性。正如我们的项目描述中所详述的，这些包括（1）度量嵌入，（2）谱分解，（3）分层表示和（4）符号表示。除了单独研究这四种表现形式中每一种的含义外，我们假设它们合在一起可以形成 一个分层的系统，作为一个整体来支持学习和行动控制的有时相互竞争的需求。 智力优势（由申请人提供）：理解表征结构如何影响学习和决策是认知科学，行为神经科学和人工智能的核心挑战。成功地建立一个计算明确的，经验验证的理论在这一领域，特别是在R中的代表性的作用，将代表一个重要的成就与广泛的影响。利用机器学习的概念工具来研究人类行为和大脑功能的策略可以提供相当大的科学杠杆作用，正如我们之前的研究所表明的那样。拟议的工作以既定的研究为动力，并以这些研究为基础，将这些研究结合在一起，以利用协同增效的机会。除了回答具体的经验和计算问题，拟议的工作旨在为未来的研究开辟新的途径，在一个重要的调查领域。 更广泛的影响（由申请人提供）：拟议的工作位于神经科学，心理学，人工智能和机器学习的十字路口，并有望推动这些领域之间日益增长的交流。该项目汇集了不同学科背景的调查人员，明确的目标是在知识分子之间架起桥梁。拟议的工作可能会找到广泛的科学受众，因为它与认知和发展心理学，行为，认知和系统神经科学以及行为经济学有关。然而，这项工作可能在人工智能、机器学习和机器人技术中同样有趣，目前的挑战是理解表征学习如何允许RL扩展到大型问题。RL的代表性方法已经在行业内引起了强烈的兴趣，目前的调查人员有积极参与的记录。拟议工作的主题也适用于其他领域，包括教育和培训以及军事和医疗决策支持。该项目的计划在研究生和博士后两级都有一个强有力的培训部分，致力于促进代表性不足的少数群体的参与以及国际参与。