CAREER: Theoretical Foundations of Offline Reinforcement Learning

职业：离线强化学习的理论基础

• 批准号: 2141781
• 负责人: Nan Jiang
• 金额: $ 50万
• 依托单位: University of Illinois at Urbana-Champaign
• 依托单位国家: 美国
• 项目类别: Continuing Grant
• 财政年份: 2022
• 资助国家: 美国
• 起止时间: 2022-05-01 至 2027-04-30
• 项目状态: 未结题
• 来源: https://www.nsf.gov/awardsearch/showAward?AWD_ID=2141781&HistoricalAwards=false
• 关键词: CAREER; Theoretical; Foundations; Offline; Reinforcement

This award is funded in whole or in part under the American Rescue Plan Act of 2021 (Public Law 117-2).Reinforcement learning (RL) is a subarea of Artificial Intelligence (AI) that solves complex decision-making tasks. It has achieved impressive successes in simulator-defined problems, where the RL agent learns via trial-and-error inside a virtual "online" environment. However, it is difficult to apply these online algorithms to real-world problems, as trial-and-error is often expensive or impossible in real life. For example, it is unethical for an RL agent in personalized medicine to test a new treatment strategy that may harm patients, just for the purpose of gathering new information. A promising paradigm to addressing this issue is offline RL, where the agent learns solely from historical data. While the lack of direct interactions with the real environment prevents undesirable real-world consequences, it also gives rise to significant technical challenges in learning. This project aims to develop novel methods to address these challenges and provide a deep theoretical understanding for offline RL, and make significant progress in enabling offline RL in real-life applications such as robotics, adaptive medical treatment, and online recommendation systems. The research development will also be integrated into the project's educational plan, which includes advising underrepresented students and developing new courses and a monograph on reinforcement learning. The technical aims of the project consist of two thrusts. The first thrust focuses on the problem of model selection: after training is completed, how should we select between candidate policies on a holdout dataset? Model selection enables hyperparameter tuning, which is the backbone of practical machine learning, yet it is notoriously difficult in offline RL due to the multi-stage nature of the problem. The proposal describes a promising approach that builds on the investigator's recent theoretical work on value-function selection. The project will devise empirically effective methods based on the theoretical insights and address practical issues such as poorly fitted candidate functions and data with insufficient coverage. The second thrust considers the theoretical foundation of offline RL training: under what conditions can we guarantee the success of training? The proposal lays out the theoretical landscape of offline-RL training, and identifies important open questions and opportunities for discovering novel theoretical and algorithmic insights.This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.

该奖项全部或部分由2021年美国救援计划法案（公法117-2）资助。强化学习（RL）是人工智能（AI）的一个子领域，用于解决复杂的决策任务。它在模拟器定义的问题中取得了令人印象深刻的成功，其中RL代理通过虚拟“在线”环境中的试错来学习。然而，很难将这些在线算法应用于现实世界的问题，因为试错法在真实的生活中通常是昂贵的或不可能的。例如，个性化医疗中的RL代理测试可能伤害患者的新治疗策略是不道德的，只是为了收集新信息。解决这个问题的一个有前途的范例是离线RL，其中代理仅从历史数据中学习。虽然缺乏与真实的环境的直接交互可以防止不期望的现实世界后果，但它也会在学习中产生重大的技术挑战。该项目旨在开发新的方法来应对这些挑战，并为离线RL提供深入的理论理解，并在现实生活中的应用中实现离线RL方面取得重大进展，如机器人，自适应医疗和在线推荐系统。研究开发也将纳入该项目的教育计划，其中包括为代表性不足的学生提供建议，开发新课程和关于强化学习的专著。该项目的技术目标包括两个方面。第一个重点是模型选择的问题：在训练完成后，我们应该如何在一个保持数据集上的候选策略之间进行选择？模型选择使超参数调整成为可能，这是实际机器学习的支柱，但由于问题的多阶段性质，离线RL非常困难。该提案描述了一个有前途的方法，建立在调查员最近的理论工作的价值功能的选择。该项目将根据理论见解设计出经验有效的方法，并解决实际问题，例如候选函数拟合不佳和覆盖范围不足的数据。第二个推力考虑离线RL训练的理论基础：在什么条件下我们可以保证训练的成功？该奖项反映了NSF的法定使命，并通过使用基金会的知识价值和更广泛的影响审查标准进行评估，被认为值得支持。

项目成果

The Optimal Approximation Factors in Misspecified Off-Policy Value Function Estimation

• DOI: 10.48550/arxiv.2307.13332
• 发表时间: 2023-07
• 作者: P. Amortila;Nan Jiang;Csaba Szepesvari

Reinforcement Learning in Low-Rank MDPs with Density Features

• DOI: 10.48550/arxiv.2302.02252
• 发表时间: 2023-02
• 期刊: ArXiv
• 作者: Audrey Huang;Jinglin Chen;Nan Jiang

Adversarial Model for Offline Reinforcement Learning

• DOI: 10.48550/arxiv.2302.11048
• 发表时间: 2023-02
• 期刊: ArXiv
• 作者: M. Bhardwaj;Tengyang Xie;Byron Boots;Nan Jiang;Ching-An Cheng

Offline Reinforcement Learning Under Value and Density-Ratio Realizability: the Power of Gaps

• DOI: 10.48550/arxiv.2203.13935
• 发表时间: 2022-03
• 作者: Jinglin Chen;Nan Jiang

A Minimax Learning Approach to Off-Policy Evaluation in Confounded Partially Observable Markov Decision Processes

• 发表时间: 2021-11
• 期刊: Applied Bionics and Biomechanics
• 影响因子: 2.2
• 作者: C. Shi;Masatoshi Uehara;Jiawei Huang;Nan Jiang