Risk-sensitive choice and reinforcement learning under uncertainty

许多现实生活中的决定都必须在面临风险的情况下做出，这种风险是由于选择情况的不确定信息造成的。至少有两种不确定性会导致决策风险：(1)决策者选择的不确定结果；(2)决策者对选择情况的不确定知识。实际上，这两种类型的不确定性都会影响决策的经济后果，从而导致风险。我们将这些类型的风险称为“经济风险”和“感知风险”。虽然在静态条件下和在没有知觉不确定性的情况下，经济风险处理的神经机制已经建立得相当好，但经济和知觉风险敏感决策策略获得的认知和神经过程却不太清楚。在拟议的项目中，我们以我们最近在开发风险敏感的马尔可夫决策过程和强化学习算法方面的计算工作为基础，研究编码经济和感知风险的认知和神经机制。在理论层面上，这需要采用风险敏感的部分可观测马尔可夫决策过程(POMDP)框架，作为人类响应时间和选择的行为模型。在实验水平上，需要在两个行为实验和一个功能磁共振实验中对该框架进行经验验证。我们将通过三个工作方案来实现这些目标。在第一个工作包中，我们将在之前的理论工作的基础上推导出新的无模型强化学习算法，该算法同时考虑了经济和感知风险。在我们的第二个工作包中，我们将测试这样一个假设，即风险敏感的POMDP框架将在捕获人类响应时间行为方面超过以前的方法。在第三个工作包中，我们将使用fMRI来识别经济和感知风险处理的神经基础。总体而言，我们的项目将把以前关于风险敏感学习和决策的工作扩展到感知不确定性的存在。

Many real-life decisions must be made in the face of risk that is due to uncertain information about the choice situation. At least two types of uncertainty can result in decision risk: (1) the uncertain consequences of the decision maker’s choice and (2) the decision maker’s uncertain knowledge about the choice situation. In effect, both types of uncertainty can impact the economic consequences of a decision and thus result in risk. We refer to these types of risk as “economic risk” and "perceptual risk". While the neural mechanisms underlying economic risk processing are fairly well established for static conditions and in the absence of perceptual uncertainty, the cognitive and neural processes underlying the acquisition of economic and perceptual risk-sensitive decision strategies are less clear. In the proposed project, we build on our recent computational work in the development of risk-sensitive Markov decision processes and reinforcement learning algorithms to investigate the cognitive and neural mechanisms of encoding economic and perceptual risk. At the theoretical level, this entails the adaptation of a risk-sensitive partially-observable Markov decision process (POMDP) framework as a behavioral model for human response times and choices. At the experimental level, it entails the empirical validation of the framework in two behavioral and one fMRI experiments. We will pursue these aims in three work packages. In the first work package, we will build on our previous theoretical work to derive novel model-free reinforcement learning algorithms that account for both economic and perceptual risk. In our second work package, we will test the hypothesis that the risk-sensitive POMDP framework will surpass previous approaches in capturing human response time behavior. In the third work package, we will use fMRI to identify the neural underpinnings of economic and perceptual risk processing. Overall, our project will extend previous work on risk-sensitive learning and decision making to the presence of perceptual uncertainty.