Anticipative Human-Robot Collaboration (P8)

有效的人机协作要求机器人系统能够观察人的行为，并对协同工作空间的未来状态进行预测，从而产生预期的机器人行为。该项目的目标是建立一个共享的人机工作空间，在多个时空尺度上预测语义上有意义的未来状态，并计划预期的机器人动作，以实现人机协作。我们将在第一阶段项目“学习预期人机协作的分层表示”中开发的协作操作的小型工作空间扩展到更大的协作移动操作场景。由于工作空间将从多个视点观察，因此必须将单个视点的语义感知和预测合并到协作场景及其变化的三维非中心语义模型中。我们将明确建模固有的传感器校准和传感器坐标帧和非中心场景帧之间的3D转换。场景将通过简单的局部3D表示来建模，比如占用网格、冲浪或距离场，以及它们的语义属性，比如类别概率，以及它们的变化。整个模型就是一个可微函数图。这些转换和表示的参数将通过即时预测单个视图来学习，给定聚合在3D表示中的其他视图。这将产生例如相机姿势估计。表征的预测方面将通过汇总过去的观点来预测未来的观点来学习。由于场景中智能体（即人类和机器人）的意图和行为具有重要的预测信息，因此对未来场景状态的预测不仅取决于当前场景状态，还取决于人类或机器人的意图和行为。这将产生在多个时空抽象层次上的人机联合工作空间的预测语义状态，这将用于以从粗到细的抽象到具体的方式规划预期的机器人行为。我们将展示我们的方法在协作移动操作任务中的实用性，其中机器人通过在正确的时刻以正确的顺序提供所需的对象来支持人类。

Effective human-robot collaboration requires the robot system to observe human actions and to make predictions on the future state of the collaborative work space, in order to generate anticipatory robot behavior. The objective of this project is to model a shared human-robot workspace, predict semantically meaningful future states on multiple spatio-temporal scales, and plan anticipatory robot actions for realizing human-robot collaboration. We will extend the small work space for collaborative manipulation developed in the first-phase project "Learning Hierarchical Representations for Anticipative Human-Robot Collaboration" to a larger collaborative mobile manipulation scenario. As the work space will be observed from multiple viewpoints, the semantic perception and prediction of individual views must be merged to a 3D allocentric semantic model of the collaborative scene and its changes. We will explicitly model the intrinsic sensor calibrations and the 3D transformations between the sensor coordinate frames and the allocentric scene frame. The scene will be modeled by simple localized 3D representations, like occupancy grids, surfels or distance fields, their semantic attributes like class probabilities, and their changes. The entire model will be a differentiable function graph. The parameters of these transformations and representations will be learned by instantaneously predicting individual views, given the other views that are aggregated in the 3D representation. This will yield e.g. camera pose estimates. The predictive aspects of the representations will be learned by predicting future views from aggregated past views. As the intentions and actions of the agents in the scene, i.e. humans and robots, have important predictive information, predictions of future scene states will be conditioned not only on the current scene state, but also on human or robot intents and actions. This yields a predicted semantic state of the joint human-robot work space on multiple levels of spatio-temporal abstraction, which will be used to plan anticipative robot behavior in a coarse-to-fine abstract-to-concrete manner. We will demonstrate the utility of our approach in collaborative mobile manipulation tasks, where the robot supports the human by providing the needed objects in the right order at the right moment.