Trustworthy Reinforcement Learning for Online Decision Making

用于在线决策的值得信赖的强化学习

• 批准号: 2217440
• 负责人: Wei Sun
• 金额: $ 45万
• 依托单位: Purdue University
• 依托单位国家: 美国
• 项目类别: Standard Grant
• 财政年份: 2022
• 资助国家: 美国
• 起止时间: 2022-08-15 至 2025-07-31
• 项目状态: 未结题
• 来源: https://www.nsf.gov/awardsearch/showAward?AWD_ID=2217440&HistoricalAwards=false
• 关键词: Trustworthy; Reinforcement; Learning; Online; Decision

This research project will advance the frontiers of modern reinforcement learning for online decision-making problems. Reinforcement learning deals with how intelligent agents ought to take actions in an uncertain environment in order to maximize the cumulative reward. It has achieved phenomenal success in diverse business and scientific fields. However, less attention has been paid to the trustworthy aspects of reinforcement learning. This project will investigate different aspects of trustworthy issues like robustness, fairness, causality, and explainability in important online decision-making tasks. The results of this research will benefit many different fields such as statistics, machine learning, operations research, marketing, economics, and finance. Open-source software will be developed to provide applied researchers with cutting-edge tools. The project will recruit students, especially those from unrepresented groups, to be involved in the research and will develop new courses on statistical reinforcement learning and decision making.This research project will focus on three interconnected trustworthy reinforcement learning methods for online decision-making problems: dynamic pricing, dynamic assortment selection, and matching in two-sided markets. Issues of robustness, fairness, causality, and explainability will be addressed in these decision-making tasks, which will advance the exploration techniques used in existing reinforcement learning algorithms. The project will develop new theoretical tools to analyze the statistical properties of these modern reinforcement learning algorithms. Regret upper bounds and matching lower bound will be thoroughly investigated. One important goal of online decision making is to identify an optimal policy that maximizes the overall gain, based on the contextual information and historical interactions with the environment. Due to the complex nature of such problems, there is a high demand for trustworthy tools for learning optimal personalized policy in various settings. The knowledge gained from this research will benefit learning in online auctions and other complex market design problems.This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.

该研究项目将推进现代强化学习在线决策问题的前沿。强化学习涉及智能体如何在不确定的环境中采取行动，以最大化累积奖励。它在不同的商业和科学领域取得了惊人的成功。然而，人们对强化学习的可信方面关注较少。该项目将研究重要在线决策任务中可信问题的不同方面，例如稳健性、公平性、因果关系和可解释性。这项研究的成果将惠及统计学、机器学习、运筹学、营销、经济学和金融等许多不同领域。将开发开源软件，为应用研究人员提供尖端工具。该项目将招募学生，特别是来自无代表性群体的学生参与研究，并将开发有关统计强化学习和决策的新课程。该研究项目将重点研究在线决策问题的三种相互关联的可信强化学习方法：动态定价、动态品种选择和双边市场匹配。这些决策任务将解决鲁棒性、公平性、因果性和可解释性问题，这将推进现有强化学习算法中使用的探索技术。该项目将开发新的理论工具来分析这些现代强化学习算法的统计特性。遗憾上限和匹配下限将被彻底调查。在线决策的一个重要目标是根据上下文信息和与环境的历史交互来确定最大化总体收益的最佳策略。由于此类问题的复杂性，对于在各种环境下学习最佳个性化策略的值得信赖的工具有很高的需求。从这项研究中获得的知识将有利于学习在线拍卖和其他复杂的市场设计问题。该奖项反映了 NSF 的法定使命，并通过使用基金会的智力价值和更广泛的影响审查标准进行评估，被认为值得支持。

项目成果

Distribution-free Contextual Dynamic Pricing

• DOI: 10.1287/moor.2023.1369
• 发表时间: 2021-09
• 期刊: Math. Oper. Res.
• 作者: Yiyun Luo;W. Sun;Yufeng Liu

Estimation of Linear Functionals in High Dimensional Linear Models: From Sparsity to Non-sparsity

高维线性模型中线性泛函的估计：从稀疏到非稀疏

• DOI: 10.1080/01621459.2023.2206084
• 发表时间: 2023
• 期刊: Journal of the American Statistical Association
• 影响因子: 3.7
• 作者: Zhao, Junlong;Zhou, Yang;Liu, Yufeng
• 通讯作者: Liu, Yufeng

Contextual Dynamic Pricing with Unknown Noise: Explore-then-UCB Strategy and Improved Regrets

未知噪声下的上下文动态定价：探索然后 UCB 策略和改进遗憾

• 发表时间: 2022
• 期刊: 36th Conference on Neural Information Processing Systems
• 作者: Yiyun Luo, Will Wei