Continuous Time Reinforcement Learning using Rough Paths

在学术和工业环境中，强化学习（RL）方法已被用于解决涉及动态系统优化的一系列科学和工程问题。这些问题涉及各个领域，包括乘车共享、医疗管理和能源系统、金融领域风险资产的定价和交易、自动驾驶汽车和机器人等领域的资源优化配置。RL也越来越多地用于物理科学，例如发现新材料和/或探索已知材料的性质。尽管RL方法的重要性和应用范围越来越广，但在现实中，RL方法的表现往往很差。强化学习方法主要是为离散时间和所谓的马尔可夫设置而开发的，而大多数现实世界的问题都是在连续时间和非马尔可夫动态中更好地建模的。RL方法在科学和工程领域的广泛采用需要研究如何开发连续时间和非马尔可夫设置的RL方法。该项目旨在为解决这些问题奠定基础。此外，pi在传播发现、研究生调查、国内和国际网络、指导初级研究人员以及研究生和本科生、参与和组织活动以及跨学科研究方面制定了具体目标。本课题的成功完成将为连续时间强化学习的理论分析做出重要贡献。该项目将提供一个适用于一般随机环境的全局框架。特别是，它超越了限制性的马尔可夫设置，并允许路径控制。该研究项目的核心是开发可用于为一系列应用领域的连续时间强化学习问题提供理论保证的分析结果。该项目的成功完成将需要一些新的结果来描述粗糙环境下路径最优控制的解决方案，分析获得最优策略的计算方法，以及分析使用粗糙路径签名近似策略和值函数的数值方案。拟议的努力将有足够的新颖性来开辟新的研究领域。它们还将进一步促进上述理论技术的适用性。该奖项反映了美国国家科学基金会的法定使命，并通过使用基金会的知识价值和更广泛的影响审查标准进行评估，被认为值得支持。

Reinforcement learning (RL) methods have been embraced, in both academic and industrial settings, for solving a range of science and engineering problems involving dynamic system optimization. These problems run the gamut and include optimal resource allocation in, for example, ride-sharing, healthcare management and energy systems, pricing and trading risky assets in finance, autonomous vehicles and robots. RL is also increasingly being used in the physical sciences, for instance for discovering new materials and/or exploring the properties of known materials. Despite the growing importance and breadth of applications, RL methods are often found to perform poorly in actuality. RL methods have been primarily developed for discrete-time and so-called Markovian settings, while most real-world problems are better modeled in continuous time, with non-Markovian dynamics. The broad adoption of RL methods across science and engineering necessitates the investigation of how to develop RL methods for continuous-time and non-Markovian settings. This project aims at laying the foundations for addressing these questions. In addition, the PIs have developed specific aims in terms of dissemination of discoveries, survey for graduate students, national and international networking, mentoring of junior researchers as well as graduate and undergraduate students, participation and organization of events, and interdisciplinary research.The successful completion of this project will fill make significant contributions towards the theoretical analysis of continuous-time RL. The project will offer a global framework valid for general random environments. In particular it goes beyond the somewhat restrictive Markov setting, and allows for pathwise controls. At its core, this research project aims at the development of analytical results that can be used to provide theoretical guarantees for continuous-time RL problems across a range of application domains. The successful completion of the project will entail a number of new results characterizing the solution of pathwise optimal control in rough environments, the analysis of computational methods for obtaining optimal policies, as well as the analysis of numerical schemes for approximating policies and value functions using rough path signatures. The proposed efforts will have sufficient novelty to open new research areas. They will also further promote the applicability of the theoretical techniques alluded to above.This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.