Risk-sensitive choice and reinforcement learning under uncertainty

不确定性下的风险敏感选择与强化学习

• 批准号: 407012307
• 负责人: Professor Dr. Klaus Obermayer
• 金额: --
• 依托单位: Institut für Psychologie
• 依托单位国家: 德国
• 项目类别: Research Grants
• 财政年份: 2018
• 资助国家: 德国
• 起止时间: 2017-12-31 至 2021-12-31
• 项目状态: 已结题
• 来源: https://gepris.dfg.de/gepris/projekt/407012307?language=en
• 关键词: Risk; sensitive; choice; reinforcement; learning

Many real-life decisions must be made in the face of risk that is due to uncertain information about the choice situation. At least two types of uncertainty can result in decision risk: (1) the uncertain consequences of the decision maker’s choice and (2) the decision maker’s uncertain knowledge about the choice situation. In effect, both types of uncertainty can impact the economic consequences of a decision and thus result in risk. We refer to these types of risk as “economic risk” and "perceptual risk". While the neural mechanisms underlying economic risk processing are fairly well established for static conditions and in the absence of perceptual uncertainty, the cognitive and neural processes underlying the acquisition of economic and perceptual risk-sensitive decision strategies are less clear. In the proposed project, we build on our recent computational work in the development of risk-sensitive Markov decision processes and reinforcement learning algorithms to investigate the cognitive and neural mechanisms of encoding economic and perceptual risk. At the theoretical level, this entails the adaptation of a risk-sensitive partially-observable Markov decision process (POMDP) framework as a behavioral model for human response times and choices. At the experimental level, it entails the empirical validation of the framework in two behavioral and one fMRI experiments. We will pursue these aims in three work packages. In the first work package, we will build on our previous theoretical work to derive novel model-free reinforcement learning algorithms that account for both economic and perceptual risk. In our second work package, we will test the hypothesis that the risk-sensitive POMDP framework will surpass previous approaches in capturing human response time behavior. In the third work package, we will use fMRI to identify the neural underpinnings of economic and perceptual risk processing. Overall, our project will extend previous work on risk-sensitive learning and decision making to the presence of perceptual uncertainty.

许多现实生活中的决策必须面对风险，这是由于不确定的信息选择的情况。至少有两种类型的不确定性会导致决策风险：（1）决策者选择的不确定后果;（2）决策者对选择情况的不确定知识。实际上，这两种类型的不确定性都可能影响决策的经济后果，从而导致风险。我们将这些风险称为“经济风险”和“感知风险”。虽然经济风险处理的神经机制是相当好的建立静态条件下，在知觉的不确定性的情况下，认知和神经过程的收购经济和知觉的风险敏感的决策策略是不太清楚。在拟议的项目中，我们建立在我们最近的计算工作，在风险敏感的马尔可夫决策过程和强化学习算法的发展，以调查编码经济和感知风险的认知和神经机制。在理论层面，这需要将风险敏感的部分可观察马尔可夫决策过程（POMDP）框架调整为人类响应时间和选择的行为模型。在实验层面，它需要在两个行为和一个功能磁共振成像实验的框架的实证验证。我们将通过三个工作包来实现这些目标。在第一个工作包中，我们将建立在我们以前的理论工作基础上，推导出新颖的无模型强化学习算法，该算法同时考虑了经济和感知风险。在我们的第二个工作包中，我们将测试风险敏感的POMDP框架将超过以前的方法在捕捉人类响应时间行为的假设。在第三个工作包中，我们将使用功能磁共振成像来识别经济和感知风险处理的神经基础。总的来说，我们的项目将把以前关于风险敏感学习和决策的工作扩展到感知不确定性的存在。