Generalization Guarantees for Imitation Learning
Generalization Guarantees for Imitation Learning
复制标题
模仿学习的泛化保证
DOI：
--
复制
发表时间：
2020
期刊：
影响因子：
--
通讯作者：
Anirudha Majumdar
中科院分区：
文献类型：
--
作者：
Allen Z. Ren;Sushant Veer;Anirudha Majumdar
由于演示不完美或模仿学习算法无法准确推断专家的策略，模仿学习的控制策略通常无法推广到新的环境。在本文中，我们提出了严格的泛化保证模仿学习，利用可能近似正确（PAC）贝叶斯框架，以提供在新的环境中的政策的预期成本的上限。我们提出了一个两阶段的训练方法，其中一个潜在的政策分布是第一次嵌入多模态专家行为使用条件变分自动编码器，然后“微调”在新的训练环境中显式优化的泛化范围。我们展示了强大的泛化边界和他们的紧密性相对于经验性能的模拟（一）抓住不同的杯子，（二）平面推动视觉反馈，（三）基于视觉的室内导航，以及通过硬件实验的两个操作任务。
Control policies from imitation learning can often fail to generalize to novel environments due to imperfect demonstrations or the inability of imitation learning algorithms to accurately infer the expert's policies. In this paper, we present rigorous generalization guarantees for imitation learning by leveraging the Probably Approximately Correct (PAC)-Bayes framework to provide upper bounds on the expected cost of policies in novel environments. We propose a two-stage training method where a latent policy distribution is first embedded with multi-modal expert behavior using a conditional variational autoencoder, and then "fine-tuned" in new training environments to explicitly optimize the generalization bound. We demonstrate strong generalization bounds and their tightness relative to empirical performance in simulation for (i) grasping diverse mugs, (ii) planar pushing with visual feedback, and (iii) vision-based indoor navigation, as well as through hardware experiments for the two manipulation tasks.