CAREER: Soft-robust Methods for Offline Reinforcement Learning

传感器、数据收集和计算能力的改进推动了利用数据来改善各个领域的决策的愿望，例如精准农业或医学。做出有效的数据驱动决策是强化学习领域研究的一个具有挑战性的问题。尽管它们在棋类游戏等模拟领域取得了成功，但现有的强化学习算法仍然太不可靠，无法广泛应用。该项目开发了新的强化学习算法，通过仔细平衡推荐决策的预期质量和失败风险来实现可靠性。失败的风险是使用随机金融的技术来衡量的，这使得算法变得灵活而高效。这些新的、可靠的算法将有助于将数据驱动的决策引入新的领域，包括农业、生态管理和医学。这项研究与教育活动相结合，为研究生和本科生提供培训机会和新的学习材料，包括一本教科书。这项研究开发和分析了强化学习(RL)问题的算法，包括1)有限或昂贵的数据和2)高失败代价。软稳健目标建立在凸性风险度量之上，以平衡数据驱动的决策问题的稳健性和平均质量。虽然软-稳健目标在单阶段优化问题中得到了很好的理解，但许多基本的和实际的问题在多阶段问题中仍然是悬而未决的，比如RL。这个项目将回答这些问题，并通过解决三个主要目标来开发可靠、易处理和可扩展的算法。首先，该项目将建立软健壮的RL公式的统计和计算特性。其次，该项目将基于对软健壮性和健壮性马尔可夫决策过程之间关系的新见解，生成一类新的表格软健壮性RL算法。第三，该项目将把表格算法扩展到值函数近似和梯度式方法。该项目将涉及批次RL和反向RL，前者具有已知的奖励和估计的转换概率，后者具有估计的奖励和已知的转换概率。该奖项反映了NSF的法定使命，并通过使用基金会的智力优势和更广泛的影响审查标准进行评估，被认为值得支持。

Improvements in sensors, data collection, and computational power have driven the desire to harness data to improve decision-making in various domains, such as precision agriculture or medicine. Making effective data-driven decisions is a challenging problem studied by the reinforcement learning community. Despite their success in simulated domains, like board games, existing reinforcement learning algorithms are still too unreliable for widespread deployment. This project develops new reinforcement learning algorithms that achieve reliability by carefully balancing the expected quality of recommended decisions with their risk of failure. The risk of failure is measured using techniques from stochastic finance, which enable flexible and efficient algorithms. These new, reliable algorithms will help bring data-driven decision-making to new domains, including agriculture, ecological management, and medicine. The research is integrated with educational activities to provide graduate and undergraduate students with training opportunities and new study materials, including a textbook. This research develops and analyzes algorithms for reinforcement learning (RL) problems with 1) limited or expensive data and 2) a high cost of failure. Soft-robust objectives build on convex risk measures to balance robustness and average quality of data-driven decision-making problems. While soft-robust objectives are well understood in single-stage optimization problems, many fundamental and practical questions remain open in multi-stage problems, like RL. This project will answer these questions and develop reliable, tractable, and scalable algorithms by tackling three main objectives. First, the project will establish soft-robust RL formulations' statistical and computational properties. Second, the project will generate a new class of tabular soft-robust RL algorithms built on new insights into the relationship between soft-robustness and robust Markov decision processes. Third, the project will scale the tabular algorithms to value function approximation and gradient style methods. The project will address both batch RL, with known rewards and estimated transition probabilities, and inverse RL, with estimated rewards and known transition probabilities.This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.