RI: Small: Towards Optimal and Adaptive Reinforcement Learning with Offline Data and Limited Adaptivity

强化学习（RL）是机器学习中发展最快的研究领域之一。基于强化学习的技术最近在人工智能领域取得了几项突破，比如在围棋比赛中击败了人类冠军。然而，RL在现实生活问题中的应用仍然有限，即使在已经收集了大量数据的领域也是如此。问题的关键在于，大多数现有的强化学习方法都需要一个代理与之交互的环境，但在现实应用中，很少有可能访问这样的环境——部署一个通过试错来学习的算法可能会有严重的法律、道德和安全问题。这个项目旨在通过开发从离线数据中学习的算法来解决这个难题。这项研究的结果可以显著减少在现实生活中使用强化学习技术解决顺序决策问题的开销，比如电力传输、个性化医疗、科学发现、计算机网络和公共政策。该项目侧重于两种环境，旨在解决上述环境有限的挑战。在第一种设置中，只向代理提供与环境交互记录的历史数据。在第二种设置中，代理只能改变它与环境的交互方式几次。研究人员将开发数学理论来描述问题的难度，并确保所开发的算法在使用尽可能少的资源（数据、能源、计算）的意义上是鲁棒和最佳的。利用边缘重要性采样、均匀收敛和批量探索等技术，该项目将推广“打破地平线诅咒”的最新工作路线，以允许函数近似，并为离线和低自适应强化学习建立急需的统计学习理论。该奖项反映了美国国家科学基金会的法定使命，并通过使用基金会的知识价值和更广泛的影响审查标准进行评估，被认为值得支持。

Reinforcement learning (RL) is one of the fastest-growing research areas in machine learning. RL-based techniques have led to several recent breakthroughs in artificial intelligence, such as beating human champions in the game of Go. The application of RL to real life problems, however, remains limited, even in areas where a large amount of data has already been collected. The crux of the problem is that most existing RL methods require an environment for the agent to interact with, but in real-life applications, it is rarely possible to have access to such an environment — deploying an algorithm that learns by trial-and-errors may have serious legal, ethical and safety issues. This project aims to address this conundrum by developing algorithms that learn from offline data. The outcome of the research could significantly reduce the overhead of using RL techniques in real-life sequential decision-making problems such as those in power transmission, personalized medicine, scientific discoveries, computer networking and public policy.The project focuses on two settings that aim at addressing the aforementioned challenge of limited access to an environment. In the first setting, the agent is given only the historical data from logged interactions with the environment. In the second setting, the agent is able to change how it interacts with the environment only a few times. The investigators will develop mathematical theory that describes the difficulty of the problem and ensures that the developed algorithms are robust and optimal in the sense that they use the least possible resources (data, energy, computation). Using techniques such as marginalized importance sampling, uniform convergence and batched exploration, the project will generalize the recent line of work in ``breaking the curse of horizon'' to allow function approximations and establish the much-needed statistical learning theory for offline and low-adaptive reinforcement learning.This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.