CRCNS: Reinforcement learning in multi-dimensional action spaces

CRCNS：多维行动空间中的强化学习

• 批准号: 8068884
• 负责人: Nathaniel Douglass Daw
• 金额: $ 37.4万
• 依托单位: NEW YORK UNIVERSITY
• 依托单位国家: 美国
• 财政年份: 2009
• 资助国家: 美国
• 起止时间: 2009-09-01 至 2014-04-30
• 项目状态: 已结题
• 来源: https://reporter.nih.gov/project-details/8068884
• 关键词: Address; Animals; Area; Behavior; Brain; Choice Behavior; Clinical; Communication; Decision Making; Decision Theory; Development; Diagnosis; Disease; Dopamine; Electrophysiology (science); Employee Strikes; Experimental Designs; Eye; Eye Movements; Functional Magnetic Resonance Imaging; Functional disorder; Generations; Hand; Harvest; Human; Joints; Laboratories; Learning; Link; Maps; Measurement; Mental disorders; Methods; Modeling; Motor; Neurons; New York; Outcome; Parietal; Parietal Lobe; Patients; Population; Procedures; Professional Education; Prosencephalon; Psychological reinforcement; Rewards; Role; Saccades; Schizophrenia; Schools; Scientist; Screening procedure; Signal Transduction; Sorting - Cell Movement; Students; System; Testing; Training; Training Programs; Translating; Work; analytical tool; clinically relevant; coping; design; dopamine system; experimental analysis; high school; improved; innovation; meetings; nervous system disorder; neural circuit; novel; outreach; programs; relating to nervous system; research study; scale up; severe mental illness; symposium; theories

DESCRIPTION (provided by applicant): A striking range of mental disorders, from OCD to schizophrenia, is accompanied by aberrant decision-making and also by dysfunction in the dopamine system and its targets in the forebrain. Although celebrated computational work posits roles for this system together with the posterior parietal cortex in learning and decision-making for simple choice problems, it requires a tremendous leap of faith to imagine how these simple computational mechanisms can be "scaled up" from the laboratory to address real-world human behavior of the sort that is clinically problematic for patients with these disorders. One understudied aspect of this problem is the high dimensionality of the space of candidate actions, notably the involvement of multiple effectors such as hands and eyes. This project proposes a theoretical framework for more realistic learning and decision problems involving multiple effectors, and leverages it in experiments probing how the brain copes with learning and decision-making in these cases. The core idea is that the brain should divide-and-conquer: treating, e.g., hand and eye movements independently to simplify learning when their consequences are independent, but that it must evaluate actions jointly across effectors when this is not the case. Learning tasks manipulating this independence are used to: (1) test whether humans and animals learn to solve decision problems by separating or coordinating effector choices to efficiently harvest rewards; these tasks are combined with electrophysiological recordings and fMRI to (2) test whether separate or conjoint neural value maps are maintained for action values across effectors, as appropriate to the problem; and multiarea recordings are used to (3) test whether coordinated choices increase neural interactions between effector-specific motor maps. The work makes innovative use of computational theory for experimental design and analysis, in order to connect experimental observations across species, measurement types (spiking, local field potentials, fMRI), and scales (neuronal, systems). It also introduces a new laboratory microcosm for the computations needed to scale up existing decision theories toward clinically relevant real-world behaviors. In principle, quantitative theories of the brain's decision and learning systems hold important promise for the numerous serious mental illnesses that center around these systems, such as improved procedures for diagnosis or screening candidate treatments. This project aims to "scale up" such theories -- which are, in practice, too simple to deliver on this promise -- toward explaining the interacting neural circuits that control realistic behaviors more like those that are problematic for patients with mental illnesses.

描述(由申请人提供)：从强迫症到精神分裂症的一系列精神障碍，伴随着异常的决策，也伴随着前脑中的多巴胺系统及其靶点的功能障碍。尽管著名的计算工作假定了这个系统与后顶叶皮质一起在学习和做出简单选择问题的决策中的作用，但想象如何从实验室“放大”这些简单的计算机制，以解决患有这些疾病的患者在临床上存在问题的那种真实世界的人类行为，需要巨大的信念飞跃。这个问题研究不足的一个方面是候选动作空间的高维性，特别是多个效应器的参与，如手和眼睛。这个项目为涉及多个效应器的更现实的学习和决策问题提出了一个理论框架，并将其用于探索大脑如何在这些情况下应对学习和决策的实验。核心思想是大脑应该分而治之：例如，当它们的结果独立时，大脑应该独立处理手和眼睛的运动，以简化学习，但当情况并非如此时，大脑必须联合评估效应器的行动。操纵这种独立性的学习任务被用来：(1)测试人类和动物是否通过分离或协调效应器选择来学习解决决策问题，以有效地获取奖励；这些任务与电生理记录和功能磁共振相结合，以(2)测试是否根据问题维持不同效应器之间的动作值的单独或联合神经值映射；以及多区域记录被用于(3)测试协调选择是否增加了效应器特定运动图之间的神经交互。 这项工作创新地利用计算理论进行实验设计和分析，以便将跨物种、测量类型(尖峰、局部场电位、功能磁共振成像)和尺度(神经元、系统)的实验观察联系起来。它还引入了一个新的实验室缩影，用于将现有决策理论扩大到临床相关的现实世界行为所需的计算。原则上，大脑决策和学习系统的量化理论为围绕这些系统的许多严重精神疾病提供了重要的希望，例如改进了诊断或筛选候选治疗的程序。这个项目旨在“放大”这些理论--实际上，这些理论太简单了，无法兑现这一承诺--解释控制现实行为的相互作用的神经回路，更像是那些对精神疾病患者有问题的行为。