Masked Trajectory Models for Prediction, Representation, and Control
Masked Trajectory Models for Prediction, Representation, and Control
复制标题
DOI：
10.48550/arxiv.2305.02968
复制
发表时间：
2023-05
期刊：
影响因子：
--
通讯作者：
Philipp Wu;Arjun Majumdar;Kevin Stone;Yixin Lin;Igor Mordatch;P. Abbeel;A. Rajeswaran
中科院分区：
文献类型：
--
作者：
Philipp Wu;Arjun Majumdar;Kevin Stone;Yixin Lin;Igor Mordatch;P. Abbeel;A. Rajeswaran
我们将蒙版轨迹模型（MTM）作为顺序决策的通用抽象引入。 MTM采用轨迹，例如状态行动序列，旨在重建以同一轨迹的随机子集为条件的轨迹。通过使用高度随机的掩盖模式进行培训，MTM可以通过简单地在推理时选择适当的掩码来学习多功能网络，这些网络可以扮演不同的角色或功能。例如，相同的MTM网络可以用作前向动力学模型，逆动力学模型，甚至是离线RL代理。通过在几个连续控制任务中进行的广泛实验，我们表明相同的MTM网络（即相同的权重）可以匹配或胜过接受上述功能的专业网络。此外，我们发现MTM学到的状态表示可以显着加速传统RL算法的学习速度。最后，在离线RL基准测试中，我们发现MTM具有专门的离线RL算法竞争，尽管MTM是一种普通的自我监督学习方法，没有任何明确的RL组件。代码可从https://github.com/facebookresearch/mtm获得
We introduce Masked Trajectory Models (MTM) as a generic abstraction for sequential decision making. MTM takes a trajectory, such as a state-action sequence, and aims to reconstruct the trajectory conditioned on random subsets of the same trajectory. By training with a highly randomized masking pattern, MTM learns versatile networks that can take on different roles or capabilities, by simply choosing appropriate masks at inference time. For example, the same MTM network can be used as a forward dynamics model, inverse dynamics model, or even an offline RL agent. Through extensive experiments in several continuous control tasks, we show that the same MTM network -- i.e. same weights -- can match or outperform specialized networks trained for the aforementioned capabilities. Additionally, we find that state representations learned by MTM can significantly accelerate the learning speed of traditional RL algorithms. Finally, in offline RL benchmarks, we find that MTM is competitive with specialized offline RL algorithms, despite MTM being a generic self-supervised learning method without any explicit RL components. Code is available at https://github.com/facebookresearch/mtm