Planning and executing robotic actions using simulated image sequences created by generative deep neural networks

机器人动作规划的常用方法依赖于符号处理。例如：AI规划需要领域知识和规划算法。类似地，规划也可以基于物理模拟，必须依赖于明确编码的编程指令。与此不同的是，这项工作的目标是创建一个基于计算机视觉的系统，该系统能够生成基于隐式模拟图像序列的机器人操作动作的短计划。为此，我们将首先使用编码器-解码器DNN（深度神经网络）确定视觉场景中不同的选定操作启示。由此产生的“示能图”，因此，表示许可的行动的先决条件。然后，我们将使用一个不同的生成DNN，它将场景作为输入，并通过“想象”一个存在启示的操作动作来创建一个新的3D输出场景。因此，这个新场景显示了如果实际执行该动作，情况将如何变化。由于这是在3D中完成的，因此可以检查所产生的场景的几何一致性，并且如果可以，则系统很可能已经达到允许的动作后置条件。 使用这个“想象的”输出场景进行下一个动作，将生成简短的规划序列，然后由我们的机器人执行。这允许严格量化真实的行动结果与“想象的”结果。这项工作的核心假设是，它应该是可以创建（短）可执行的计划完全依赖于子符号信息。这样做的主要好处是，服务机器人的简单日常任务可能变得可规划，而显式表示的工作量大大减少。

Common methods for robotic action planning rely on symbolic processing. For example: AI planning requires domain knowledge and planning algorithms. Similarly, physics simulations, on which planning can be based too, have to rely on explicitly encoded programming instructions. Different from this, the goal of this work is to create a computer vision-based system that is able to generate short plans for robotic manipulation actions based on implicitly simulated image sequences. To this end, we will first determine different selected manipulation affordances in a visual scene using an encoder-decoder DNN (deep neural network). The resulting ‘affordance map’, thus, represents permissive action preconditions. Then we will use a different, generative DNN, which takes the scene as input and creates a new 3D-output scene by “imagining” one of the manipulation actions for which an affordance exists. Hence, this new scene shows how the situation would change if this action would actually be performed. As this is done in 3D, the resulting scene can be checked for geometric consistency and – if ok – the system has very likely arrived at a permitted action post-condition. Using this ‘imagined’ output scene for a next action, short planning sequences will be generated, which are then executed with our robots. This allows for rigorous quantification of real action outcome against “imagined” outcome. The central hypothesis that underlies this work is that it should be possible to create (short) executable plans relying entirely on sub-symbolic information. The main benefit of this is that simple everyday tasks for service robots might become plannable with a much reduced effort for explicit representations.