Distance Weighted Supervised Learning for Offline Interaction Data
Distance Weighted Supervised Learning for Offline Interaction Data
复制标题
DOI：
10.48550/arxiv.2304.13774
复制
发表时间：
2023-04
期刊：
影响因子：
--
通讯作者：
Joey Hejna;Jensen Gao;Dorsa Sadigh
中科院分区：
文献类型：
--
作者：
Joey Hejna;Jensen Gao;Dorsa Sadigh
顺序决策算法通常难以利用不同来源的非结构化离线交互数据。基于监督学习的模仿学习（IL）方法具有很强的鲁棒性，但需要最优的证明，这是很难收集的。离线目标条件强化学习（RL）算法承诺从次优数据中学习，但面临优化挑战，特别是高维数据。为了弥合IL和RL之间的差距，我们引入了距离加权监督学习（DWSL），这是一种用于从离线数据中学习目标条件策略的监督方法。DWSL仅通过监督学习对离线数据中状态之间的时间步长的整个分布进行建模，并使用此分布来近似最短路径距离。为了提取策略，我们通过距离估计的减少来对动作进行加权。理论上，DWSL收敛到一个最优的政策约束的数据分布，离线学习的一个有吸引力的属性，没有任何引导。在我们测试的所有数据集中，DWSL根据经验将行为克隆保持为下限，同时仍表现出策略改进。在高维图像域中，DWSL超越了先前的目标条件IL和RL算法的性能。可视化和代码可以在https://sites.google.com/view/dwsl/home上找到。
Sequential decision making algorithms often struggle to leverage different sources of unstructured offline interaction data. Imitation learning (IL) methods based on supervised learning are robust, but require optimal demonstrations, which are hard to collect. Offline goal-conditioned reinforcement learning (RL) algorithms promise to learn from sub-optimal data, but face optimization challenges especially with high-dimensional data. To bridge the gap between IL and RL, we introduce Distance Weighted Supervised Learning or DWSL, a supervised method for learning goal-conditioned policies from offline data. DWSL models the entire distribution of time-steps between states in offline data with only supervised learning, and uses this distribution to approximate shortest path distances. To extract a policy, we weight actions by their reduction in distance estimates. Theoretically, DWSL converges to an optimal policy constrained to the data distribution, an attractive property for offline learning, without any bootstrapping. Across all datasets we test, DWSL empirically maintains behavior cloning as a lower bound while still exhibiting policy improvement. In high-dimensional image domains, DWSL surpasses the performance of both prior goal-conditioned IL and RL algorithms. Visualizations and code can be found at https://sites.google.com/view/dwsl/home .