ITR: Risk, Reward, and Reinforcement

该项目的目标是开发高效和可靠的直接强化算法，学习具有高度不确定性的问题的风险厌恶行为，并将开发的方法应用于一个重要的经济问题：全球资产配置。强化学习(RL)使目标导向的智能体能够在有限的反馈下通过试错探索发现策略。直接强化(DR，或“策略梯度”)方法使代理能够在不需要学习值函数的情况下发现策略。动态规划和相关的值函数RL方法经常被发现效率低下，产生不稳定的解，并且难以扩展到大型问题。因此，值函数类型RL在现实世界中的应用相对较少。这个项目寻求在直接强化方面取得一些进展，使开发高效和有效的实际应用程序成为可能。通过控制在线学习过程中的“探索和利用”的权衡，DR代理将能够发现更好的策略，并更有效地执行这项工作。随机优化方法，如随机“搜索然后收敛”或玻尔兹曼温度的退火法是候选方法。通过开发规避风险的加固方法，灾难恢复代理将能够学习针对不确定或高风险环境的稳健政策。使用对风险敏感的跨期实用程序，DR代理将学会在追求长期回报的同时避免风险状态或行动。动态规划在经济学和金融学中得到了广泛的应用，但用强化学习来解决重要的金融问题的尝试还很少。作为风险厌恶DR的一个示范，该项目将构建一个全球资产配置系统的原型。风险厌恶直接强化可能在从机器人到工业控制再到自主代理的各种工程领域得到应用。许多行业，如能源和航空公司，需要同时管理运营和财务风险，以避免供应短缺或破产。个人投资者必须管理风险，同时建立他们的投资组合，以满足未来的需求，如子女的大学费用或退休。风险厌恶的直接增援可能会在许多这样的情况下得到应用。

The objectives of this project are to develop efficient and reliable algorithms for direct reinforcement, to learn risk-averse behaviors for problems with high degrees of uncertainty, and to apply the methods developed to an economically important problem: global asset allocation. Reinforcement learning (RL) enables a goal-directed agent to discover strategies through trial and error exploration with only limited feedback. Direct Reinforcement (DR, or "policy gradient") methods enable an agent to discover a strategy without the need to learn a value function.Dynamic programming and related value function RL methods are often found to be inefficient, to produce unstable solutions, and to have difficulty scaling up to large problems. Hence, there have been relatively few real-world applications of the value function type RL. This project seeks to make several advancements in Direct Reinforcement that will enable the development of efficient and effective practical applications.By controlling the "exploration vs. exploitation" trade-off during on-line learning, DR agents will be able to discover better policies and do so more efficiently. Stochastic optimization methods, such as stochastic "search then converge" or annealing of a Boltzmann temperature are candidate approaches. By developing risk-averse reinforcement methods, DR agents will be able to learn robust policies for uncertain or risky environments. Using risk-sensitive intertemporal utilities, DR agents will learn to avoid risky states or actions while they pursue long-term reward. Dynamic programming is widely used in economics and finance, but few attempts have been made to solve important financial problems with reinforcement learning. As a demonstration of risk-averse DR, this project will build a prototype global asset allocation system.Risk-averse direct reinforcement may find application in a variety of engineering domains, from robotics to industrial control to autonomous agents. Many industries, such as energy and the airlines, need to manage operational and financial risks together, in order to avoid supply shortfalls or bankruptcy. Individual investors must manage risk while building their investment portfolios to meet future needs, such as children's college expenses or retirement. Risk-averse Direct Reinforcement may find application in many such contexts.