Goal-driven dimensionality reduction for reinforcement learning
Goal-driven dimensionality reduction for reinforcement learning
复制标题
DOI：
10.1109/iros.2017.8206334
复制
发表时间：
2017-09
期刊：
影响因子：
--
通讯作者：
Simone Parisi;Simon Ramstedt;Jan Peters
中科院分区：
文献类型：
--
作者：
Simone Parisi;Simon Ramstedt;Jan Peters
定义一种状态表示，使最优控制能够很好地发挥作用，这是一个乏味但至关重要的过程。它通常需要专业知识，不能直接推广到不同的任务，并且强烈影响学习控制器的质量。在本文中，我们提出了一种基于目标相关特征的低维流形的自主特征构建方法，并结合最优控制器使用强化学习进行学习。我们的方法将信息论算法与主成分分析相结合，对状态表示进行回报加权约简。该方法不需要对数据进行任何预处理，对状态表示没有很强的限制，并且通过减少所需的样本数量大大提高了学习性能。我们表明，我们的方法可以在冗余空间中学习高质量的控制器，甚至可以从像素中学习，并且优于经典和最先进的深度学习方法。
Defining a state representation on which optimal control can perform well is a tedious but crucial process. It typically requires expert knowledge, does not generalize straightforwardly over different tasks and strongly influences the quality of the learned controller. In this paper, we present an autonomous feature construction method for learning low-dimensional manifolds of goal-relevant features jointly with an optimal controller using reinforcement learning. Our method combines information-theoretic algorithms with principal component analysis to performs a return-weighted reduction of the state representation. The method does not require any preprocessing of the data, does not assume strong restrictions on the state representation, and substantially improves the performance of learning by reducing the number of samples required. We show that our method can learn high quality controller in redundant spaces, even from pixels, and outperforms both classical and state-of-the-art deep learning approaches.