Robot learning to perceive, plan, and act under uncertainty

未来的机器人将需要能够规划它们的行动，以便它们能够了解环境以完成任务。这种规划在非结构化的部分可观察的真实的世界环境中尤其重要，这些环境包括家用机器人、自适应制造、老年人护理、处理危险材料，甚至是福岛等灾难场景。在这样的应用中，机器人必须依赖于多种模态，例如相机图像、激光测距仪甚至触觉和声学反馈，即使有完美的视觉传感器，机器人也无法通过遮挡观察。能够在这种环境中运行并交互式感知世界的机器人需要数据驱动的强化学习方法，这些方法可以将不确定性纳入决策过程，以主动采取信息收集行动。要做到这一点，机器人还需要发展对物理世界的基本理解，并将这些信息纳入其推理过程。该项目旨在通过以下创新使部分可观察机器人任务中的无模型强化学习变得可行：（i）我们将研究新的概率结构化记忆表示，使我们能够有效地重用不同类型策略的经验。(ii)部分可观察性下的政策学习需要信息收集行动，这些行动需要在长期范围内传播价值观，以及探索，这可以揭示这些价值观。为了实现长期的行动选择，我们将利用基于模型的方法来进行有效的探索和价值传播。(iii)在部分可观察的设置中，赋值问题被放大了。我们将遵循一种引导式强化学习方法：我们在离线策略学习过程中使用额外的辅助信息，但在在线操作过程中只使用本地感官信息。我们将通过赋予机器人扮演天皇的能力来评估这些方法上的进步。扮演天皇是一个具有挑战性的机器人操作问题，它展示了与上述部分可观测性相关的所有困难。机器人必须处理遮挡和部分信息。它必须主动测试物理特性以及哪些接触对于某些棒是活跃的，并将这些知识整合到其精细的操作技能中，以从堆中移除棒。

Future robots will need to be able to plan their actions so that they can learn about the environment in order to accomplish their tasks. This kind of planning is especially important in unstructured partially observable real world environments found in household robotics, adaptive manufacturing, elderly care, handling dangerous materials, or even disaster scenarios such as Fukujima. In such applications, the robot has to rely on multiple modalities such as camera images, laser range finders or even tactile and acoustic feedback and even with perfect visual sensors the robot cannot observe through occlusions. Robots that can operate in such environments and interactively perceive the world, need data-driven reinforcement learning methods that can incorporate uncertainty in the decision making process to proactively take information gathering actions. To do so, the robot also needs to develop a basic understanding of the physical world and incorporate this information into its reasoning process. This project aims to make model-free reinforcement learning in partially observable robotic tasks feasible through following innovations: (i) We will investigate new probabilistic structured memory representations which allow us to efficiently reuse experience with different kinds of policies. (ii) Policy learning under partial observability requires information gathering actions which need propagation of values over long horizons, and exploration, which can uncover those values. To enable long-term action selection, we will utilize ideas from model-based methods for efficient exploration and value propagation. (iii) In partially observable settings the value assignment problem is amplified. We will follow a guided reinforcement learning approach: we use additional side information during offline policy learning but use only local sensory information during online operation.We will evaluate these methodological advances by endowing a robot with the ability to play Mikado. Playing Mikado is a challenging robotic manipulation problem that exhibits all the difficulties connected to partial observability described above. The robot has to deal with occlusions and partial information. It has to proactively test physical properties and which contacts are active for certain sticks and integrate this knowledge into its fine manipulation skills to remove sticks from the heap.