Provably Efficient Offline Reinforcement Learning with Perturbed Data Sources
Provably Efficient Offline Reinforcement Learning with Perturbed Data Sources
复制标题
DOI：
10.48550/arxiv.2306.08364
复制
发表时间：
2023-06
期刊：
影响因子：
--
通讯作者：
Chengshuai Shi;Wei Xiong;Cong Shen;Jing Yang
中科院分区：
文献类型：
--
作者：
Chengshuai Shi;Wei Xiong;Cong Shen;Jing Yang
现有的离线强化学习（RL）理论研究大多考虑直接从目标任务中采样的数据集。然而，在实践中，数据往往来自几个不同但相关的来源。受这一差距的影响，这项工作的目的是严格理解离线RL与多个数据集，这些数据集是从目标任务的随机扰动版本而不是从其本身收集的。一个信息理论的下限推导，这揭示了一个必要的要求，除了对数据样本的数量所涉及的源的数量。然后，提出了一种新的HetPEI算法，该算法同时考虑了每个数据源有限数量的数据样本的样本不确定性和由于有限数量的可用数据源的源不确定性。理论分析表明，只要数据源共同提供良好的数据覆盖率，HetPEI可以解决目标任务。此外，HetPEI被证明是最佳的多项式因子的地平线长度。最后，研究扩展到离线马尔可夫博弈和离线鲁棒RL，这表明所提出的设计和理论分析的通用性。
Existing theoretical studies on offline reinforcement learning (RL) mostly consider a dataset sampled directly from the target task. In practice, however, data often come from several heterogeneous but related sources. Motivated by this gap, this work aims at rigorously understanding offline RL with multiple datasets that are collected from randomly perturbed versions of the target task instead of from itself. An information-theoretic lower bound is derived, which reveals a necessary requirement on the number of involved sources in addition to that on the number of data samples. Then, a novel HetPEVI algorithm is proposed, which simultaneously considers the sample uncertainties from a finite number of data samples per data source and the source uncertainties due to a finite number of available data sources. Theoretical analyses demonstrate that HetPEVI can solve the target task as long as the data sources collectively provide a good data coverage. Moreover, HetPEVI is demonstrated to be optimal up to a polynomial factor of the horizon length. Finally, the study is extended to offline Markov games and offline robust RL, which demonstrates the generality of the proposed designs and theoretical analyses.