CRCNS: Reinforcement learning in multi-dimensional action spaces

CRCNS：多维行动空间中的强化学习

基本信息

批准号：
8068884
负责人：
Nathaniel Douglass Daw
金额：
$ 37.4万
依托单位：
NEW YORK UNIVERSITY
依托单位国家：
美国
项目类别：
财政年份：
2009
资助国家：
美国
起止时间：
2009-09-01 至 2014-04-30
项目状态：
已结题

来源：
https://reporter.nih.gov/project-details/8068884
关键词：
Address Animals Area Behavior Brain Choice Behavior Clinical Communication Decision Making Decision Theory Development Diagnosis Disease Dopamine Electrophysiology (science)Employee Strikes Experimental Designs Eye Eye Movements Functional Magnetic Resonance Imaging Functional disorder Generations Hand Harvest Human Joints Laboratories Learning Link Maps Measurement Mental disorders Methods Modeling Motor Neurons New York Outcome Parietal Parietal Lobe Patients Population Procedures Professional Education Prosencephalon Psychological reinforcement Rewards Role Saccades Schizophrenia Schools Scientist Screening procedure Signal Transduction Sorting - Cell Movement Students System Testing Training Training Programs Translating Work analytical tool clinically relevant coping design dopamine system experimental analysis high school improved innovation meetings nervous system disorder neural circuit novel outreach programs relating to nervous system research study scale up severe mental illness symposium theories

项目摘要

DESCRIPTION (provided by applicant): A striking range of mental disorders, from OCD to schizophrenia, is accompanied by aberrant decision-making and also by dysfunction in the dopamine system and its targets in the forebrain. Although celebrated computational work posits roles for this system together with the posterior parietal cortex in learning and decision-making for simple choice problems, it requires a tremendous leap of faith to imagine how these simple computational mechanisms can be "scaled up" from the laboratory to address real-world human behavior of the sort that is clinically problematic for patients with these disorders. One understudied aspect of this problem is the high dimensionality of the space of candidate actions, notably the involvement of multiple effectors such as hands and eyes. This project proposes a theoretical framework for more realistic learning and decision problems involving multiple effectors, and leverages it in experiments probing how the brain copes with learning and decision-making in these cases. The core idea is that the brain should divide-and-conquer: treating, e.g., hand and eye movements independently to simplify learning when their consequences are independent, but that it must evaluate actions jointly across effectors when this is not the case. Learning tasks manipulating this independence are used to: (1) test whether humans and animals learn to solve decision problems by separating or coordinating effector choices to efficiently harvest rewards; these tasks are combined with electrophysiological recordings and fMRI to (2) test whether separate or conjoint neural value maps are maintained for action values across effectors, as appropriate to the problem; and multiarea recordings are used to (3) test whether coordinated choices increase neural interactions between effector-specific motor maps. The work makes innovative use of computational theory for experimental design and analysis, in order to connect experimental observations across species, measurement types (spiking, local field potentials, fMRI), and scales (neuronal, systems). It also introduces a new laboratory microcosm for the computations needed to scale up existing decision theories toward clinically relevant real-world behaviors. In principle, quantitative theories of the brain's decision and learning systems hold important promise for the numerous serious mental illnesses that center around these systems, such as improved procedures for diagnosis or screening candidate treatments. This project aims to "scale up" such theories -- which are, in practice, too simple to deliver on this promise -- toward explaining the interacting neural circuits that control realistic behaviors more like those that are problematic for patients with mental illnesses.

描述（由申请人提供）：从强迫症到精神分裂症的惊人精神疾病范围伴随着异常的决策，以及多巴胺系统中的功能障碍及其在前脑中的目标。尽管著名的计算工作在学习和决策中为简单选择问题提供了该系统的角色，但它需要一个巨大的信念飞跃，以便想象如何将这些简单的计算机制从实验室“缩放”，从实验室“缩放”，以解决这些疾病患者临床问题的现实世界人类行为。这个问题的一个研究者是候选行动空间的高维度，特别是诸如手和眼睛等多个效应子的参与。该项目为更现实的学习和涉及多个效应子的决策问题提供了一个理论框架，并在这些情况下大脑如何应对学习和决策的实验中利用它。核心思想是，大脑应该分裂和互动：例如，治疗，手和眼睛的动作在后果独立时简化学习，但在情况并非如此时，它必须在效应方面共同评估行动。学习操纵这种独立性的任务用于：（1）测试人和动物是否通过分开或协调效应子选择来有效收获奖励来解决决策问题；这些任务与电生理记录和fMRI结合到（2）测试是否适合问题的效应子进行动作值，以确保单独或联合神经值图。和多环记录用于（3）测试协调的选择是否会增加效应特异性运动图之间的神经相互作用。这项工作使计算理论用于实验设计和分析，以连接跨物种，测量类型（尖峰，局部现场电位，fMRI）和量表（神经元，系统）的实验观测。它还引入了一个新的实验室缩影，以用于将现有决策理论扩展到临床相关的现实世界行为所需的计算。原则上，大脑决策和学习系统的定量理论对围绕这些系统的众多严重的精神疾病具有重要的希望，例如改进的诊断或筛查候选治疗方法。该项目旨在“扩展”此类理论（实际上，这些理论太简单，无法实现这一诺言）来解释相互作用的神经回路，这些神经回路控制现实的行为，更像是那些对精神疾病患者有问题的行为。