Risk-sensitive choice and reinforcement learning under uncertainty

许多现实生活中的决策都必须面对风险，而这种风险是由关于选择情况的不确定信息造成的。至少有两种类型的不确定性会导致决策风险：(1)决策者选择的不确定性后果和(2)决策者对选择情况的不确定性知识。实际上，这两种类型的不确定性都可能影响决策的经济后果，从而导致风险。我们将这些类型的风险称为“经济风险”和“感知风险”。虽然在静态条件和缺乏感知不确定性的情况下，潜在的经济风险处理的神经机制已经相当完善，但潜在的经济和感知风险敏感决策策略的认知和神经过程尚不清楚。在拟议的项目中，我们以我们最近在风险敏感马尔可夫决策过程和强化学习算法开发中的计算工作为基础，研究编码经济和感知风险的认知和神经机制。在理论层面，这需要适应风险敏感的部分可观察马尔可夫决策过程（POMDP）框架作为人类反应时间和选择的行为模型。在实验层面，它需要在两个行为和一个功能磁共振成像实验中对框架进行实证验证。我们将在三个工作包中实现这些目标。在第一个工作包中，我们将以之前的理论工作为基础，推导出考虑经济风险和感知风险的新型无模型强化学习算法。在我们的第二个工作包中，我们将测试风险敏感的POMDP框架在捕获人类响应时间行为方面将超越以前的方法的假设。在第三个工作包中，我们将使用功能磁共振成像来识别经济和感知风险处理的神经基础。总的来说，我们的项目将把以前关于风险敏感学习和决策制定的工作扩展到感知不确定性的存在。

Many real-life decisions must be made in the face of risk that is due to uncertain information about the choice situation. At least two types of uncertainty can result in decision risk: (1) the uncertain consequences of the decision maker’s choice and (2) the decision maker’s uncertain knowledge about the choice situation. In effect, both types of uncertainty can impact the economic consequences of a decision and thus result in risk. We refer to these types of risk as “economic risk” and "perceptual risk". While the neural mechanisms underlying economic risk processing are fairly well established for static conditions and in the absence of perceptual uncertainty, the cognitive and neural processes underlying the acquisition of economic and perceptual risk-sensitive decision strategies are less clear. In the proposed project, we build on our recent computational work in the development of risk-sensitive Markov decision processes and reinforcement learning algorithms to investigate the cognitive and neural mechanisms of encoding economic and perceptual risk. At the theoretical level, this entails the adaptation of a risk-sensitive partially-observable Markov decision process (POMDP) framework as a behavioral model for human response times and choices. At the experimental level, it entails the empirical validation of the framework in two behavioral and one fMRI experiments. We will pursue these aims in three work packages. In the first work package, we will build on our previous theoretical work to derive novel model-free reinforcement learning algorithms that account for both economic and perceptual risk. In our second work package, we will test the hypothesis that the risk-sensitive POMDP framework will surpass previous approaches in capturing human response time behavior. In the third work package, we will use fMRI to identify the neural underpinnings of economic and perceptual risk processing. Overall, our project will extend previous work on risk-sensitive learning and decision making to the presence of perceptual uncertainty.