Semantic Video Prediction (P6)

预测未来的测量是智能系统的一个关键能力。它可以以自我监督的方式学习，但需要发现合适的场景表示才能成功。有效的人机协作需要一个系统来观察人类的行为，并对协作工作空间的未来状态做出预测。这个项目的目标是学习一系列的表示共享的人类机器人工作空间越来越抽象，并允许增加的时间范围的预测。由于运动分割有助于预测，因此在第一阶段项目“预期人机协作的学习分层表示”中开发的分层表示的无监督学习框架应扩展到将场景分割为单个对象和人。将开发一个网络架构，将场景建模为相互作用和相互遮挡的连贯移动片段。由于未来往往有多个合理的发展，预测学习框架应扩展到明确考虑未来状态的多模态分布。为此，语义上有意义的潜在变量将被学习，多模态的未来将是有条件的-而不需要明确的标签。为了使这些表示针对人机协作，我们将对其进行微调，以实现多个未来的语义感知和语义预测。与时空分辨率相对应，更粗的语义概念，如更大的对象和更长期的活动，将在更高的层次上预测更长的未来。学习的场景表示和预测将作为项目P8“预期人机协作”的基础。

Prediction of future measurements is a key capability of intelligent systems. It can be learned in a self-supervised way but needs to discover suitable scene representations in order to be successful. Effective human-robot collaboration requires a system to observe human actions and to make predictions on the future state of the collaborative work space. The objective of this project is to learn a sequence of representations of the shared human-robot workspace which are increasingly abstract and which allow for predictions for increasing time horizons. As motion segmentation helps prediction, the framework for unsupervised learning of hierarchical representations that has been developed in the first-phase project "Learning Hierarchical Representations for Anticipative Human-Robot Collaboration" shall be extended to account for segmentation of the scene into individual objects and persons. A network architecture will be developed that models the scene as coherently moving segments that interact and occlude each other. As the future has often multiple plausible developments, the prediction learning framework shall be extended to explicitly account for multimodal distributions of future states. To this end, semantically meaningful latent variables will be learned on which the multimodal future will be conditioned -- without requiring explicit labels. To make the representations targeted to human-robot collaboration, we will fine-tune them for semantic perception and semantic prediction of multiple futures. In correspondence with the spatio-temporal resolutions, coarser semantic concepts, such as larger objects and longer-term activities shall be predicted at the higher layers longer into the future. The learned scene representations and predictions will serve as basis for project P8 "Anticipative Human-Robot Collaboration".