CAREER: Robotic Manipulation Using Deep Deictic Reinforcement Learning

随着机器人的任务和环境变得越来越复杂，用手明确地为机器人行为的每一个细节编程变得越来越具有挑战性。另一种方法是通过经验来学习行为，这是一种被称为“强化学习”的机器学习，机器人通过尝试和错误来学习。然而，单纯的试错是低效的，这意味着机器人需要很长时间来学习。该项目的目标是使机器人能够将注意力集中在环境的部分，从而有效地学习和良好地概括新任务。这项研究的结果是家庭辅助机器人的能力，例如配备机械臂的辅助轮椅，可以学习如何更好地帮助体弱多病和残疾人。该项目将开发一种新方法，通过结合指示表示将深度强化学习（deep RL）应用于机器人操作问题。指示表示对状态/动作进行编码，这些状态/动作相对于代理在环境中放置的标记。在这个项目中，标记是一个6自由度的参考系，放置在一个三维点云中，或者是截断的符号距离函数。机器人通过使用深度强化学习解决马尔可夫决策过程来决定在哪里放置标记以及相对于该标记它应该如何移动。初步结果表明，这种新方法可以使机器人学习控制策略，解决复杂的操作问题，而不需要精确的被操作对象的几何模型。虽然该方法仍然隐式地估计对象姿态的一些元素，但它以一种很好地推广到新对象的方式这样做，并且除非任务要求，否则不一定估计完整的对象姿态。该奖项反映了美国国家科学基金会的法定使命，并通过使用基金会的知识价值和更广泛的影响审查标准进行评估，被认为值得支持。

As robot tasks and environments become more complex, it is getting too challenging to program every detail of the robot's behavior explicitly, by hand. An alternate approach is to learn behaviors through experience, a type of machine learning known as ``reinforcement learning'', where the robot learns through trial and error. Pure trial and error, however, is inefficient, which means it takes the robot a long time to learn. The goal of this project is to enable robots to focus attention on the parts of the environment that lead to effective learning and good generalization to new tasks. A result of this research is the ability of assistive robots in the home, such as an assistive wheelchair equipped with a robotic arm, to learn how to better help the infirm and people with disabilities.This project will develop a new approach to applying deep reinforcement learning (deep RL) to robotic manipulation problems by incorporating deictic representations. A deictic representation encodes state/action relative to a marker that the agent places in the environment. In this project, the marker is a 6-DOF reference frame, placed in a 3-D point cloud, or truncated signed distance function. The robot decides where to place the marker and how it should move relative to that marker by solving a Markov decision process using deep reinforcement learning. Preliminary results suggest that this new method can enable robots to learn control policies that solve complex manipulation problems without the need for precise geometric models of the objects being manipulated. While the method still estimates some elements of object pose implicitly, it does so in a way that generalizes well to novel objects and does not necessarily estimate full object pose unless required by the task.This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.