CAREER: Soft-robust Methods for Offline Reinforcement Learning

传感器、数据收集和计算能力的改进促使人们渴望利用数据来改善各个领域的决策，比如精准农业或医学。做出有效的数据驱动决策是强化学习社区研究的一个具有挑战性的问题。尽管它们在模拟领域取得了成功，比如棋盘游戏，但现有的强化学习算法仍然太不可靠，无法广泛部署。该项目开发了新的强化学习算法，通过仔细平衡推荐决策的预期质量和失败风险来实现可靠性。失败的风险是使用随机金融技术来衡量的，这使得算法灵活有效。这些新的、可靠的算法将有助于将数据驱动的决策引入新的领域，包括农业、生态管理和医学。该研究与教育活动相结合，为研究生和本科生提供培训机会和新的学习材料，包括一本教科书。本研究开发和分析了1)有限或昂贵的数据和2)高失败成本的强化学习（RL）问题的算法。软鲁棒目标建立在凸风险度量上，以平衡鲁棒性和数据驱动决策问题的平均质量。虽然软鲁棒目标在单阶段优化问题中得到了很好的理解，但在多阶段问题（如强化学习）中仍有许多基本和实际问题有待解决。该项目将通过解决三个主要目标来回答这些问题，并开发可靠、易于处理和可扩展的算法。首先，该项目将建立软鲁棒强化学习公式的统计和计算特性。其次，该项目将基于对软鲁棒性和鲁棒马尔可夫决策过程之间关系的新见解，生成一类新的表格式软鲁棒强化学习算法。第三，该项目将表格算法扩展到值函数近似和梯度风格方法。该项目将处理批RL，具有已知的奖励和估计的转移概率，以及逆RL，具有估计的奖励和已知的转移概率。该奖项反映了美国国家科学基金会的法定使命，并通过使用基金会的知识价值和更广泛的影响审查标准进行评估，被认为值得支持。

Improvements in sensors, data collection, and computational power have driven the desire to harness data to improve decision-making in various domains, such as precision agriculture or medicine. Making effective data-driven decisions is a challenging problem studied by the reinforcement learning community. Despite their success in simulated domains, like board games, existing reinforcement learning algorithms are still too unreliable for widespread deployment. This project develops new reinforcement learning algorithms that achieve reliability by carefully balancing the expected quality of recommended decisions with their risk of failure. The risk of failure is measured using techniques from stochastic finance, which enable flexible and efficient algorithms. These new, reliable algorithms will help bring data-driven decision-making to new domains, including agriculture, ecological management, and medicine. The research is integrated with educational activities to provide graduate and undergraduate students with training opportunities and new study materials, including a textbook. This research develops and analyzes algorithms for reinforcement learning (RL) problems with 1) limited or expensive data and 2) a high cost of failure. Soft-robust objectives build on convex risk measures to balance robustness and average quality of data-driven decision-making problems. While soft-robust objectives are well understood in single-stage optimization problems, many fundamental and practical questions remain open in multi-stage problems, like RL. This project will answer these questions and develop reliable, tractable, and scalable algorithms by tackling three main objectives. First, the project will establish soft-robust RL formulations' statistical and computational properties. Second, the project will generate a new class of tabular soft-robust RL algorithms built on new insights into the relationship between soft-robustness and robust Markov decision processes. Third, the project will scale the tabular algorithms to value function approximation and gradient style methods. The project will address both batch RL, with known rewards and estimated transition probabilities, and inverse RL, with estimated rewards and known transition probabilities.This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.