Semantic Video Prediction (P6)

对未来测量值的预测是智能系统的一项关键能力。它可以通过自我监督的方式学习，但需要找到合适的场景表示才能成功。有效的人-机器人协作需要一个系统来观察人类的行为，并对协作工作空间的未来状态做出预测。这个项目的目标是学习人类-机器人共享工作空间的一系列表示，这些表示越来越抽象，并允许预测增加的时间范围。由于运动分割有助于预测，在第一阶段项目“学习用于预期的人-机器人协作的分层表示”中开发的分层表示的无监督学习框架应被扩展以考虑将场景分割成单独的对象和人。将开发一种网络体系结构，将场景建模为相互作用和遮挡的连贯移动的片段。由于未来往往有多个看似合理的发展，预测学习框架应扩展到明确考虑未来状态的多模式分布。为此，将学习语义上有意义的潜在变量，多模式的未来将以这些变量为条件--而不需要明确的标签。为了使表示面向人-机器人协作，我们将对它们进行微调，以实现对多个未来的语义感知和语义预测。与时空分辨率相对应，更粗略的语义概念，如更大的对象和更长期的活动，应该在更高的层预测到更长的未来。学习到的场景表示和预测将作为项目P8“预期性人-机器人协作”的基础。

Prediction of future measurements is a key capability of intelligent systems. It can be learned in a self-supervised way but needs to discover suitable scene representations in order to be successful. Effective human-robot collaboration requires a system to observe human actions and to make predictions on the future state of the collaborative work space. The objective of this project is to learn a sequence of representations of the shared human-robot workspace which are increasingly abstract and which allow for predictions for increasing time horizons. As motion segmentation helps prediction, the framework for unsupervised learning of hierarchical representations that has been developed in the first-phase project "Learning Hierarchical Representations for Anticipative Human-Robot Collaboration" shall be extended to account for segmentation of the scene into individual objects and persons. A network architecture will be developed that models the scene as coherently moving segments that interact and occlude each other. As the future has often multiple plausible developments, the prediction learning framework shall be extended to explicitly account for multimodal distributions of future states. To this end, semantically meaningful latent variables will be learned on which the multimodal future will be conditioned -- without requiring explicit labels. To make the representations targeted to human-robot collaboration, we will fine-tune them for semantic perception and semantic prediction of multiple futures. In correspondence with the spatio-temporal resolutions, coarser semantic concepts, such as larger objects and longer-term activities shall be predicted at the higher layers longer into the future. The learned scene representations and predictions will serve as basis for project P8 "Anticipative Human-Robot Collaboration".