融合视频时空语义关系的动态虚拟场景自动生成方法研究

动态虚拟场景自动生成是虚拟现实的重要研究方向，涉及场景语义理解、人与环境互动关系分析、互动场景序列生成等技术。近年来，深度学习方法在虚拟现实领域取得了突破性进展，但仍存在场景理解模型对数据依赖性高、生成虚拟人体语义细节缺失、动作序列可扩展性差的瓶颈。本项目利用时空语义关系中承载的常识知识来驱动动态虚拟场景自动生成，以知识图与场景图的双向信息传递、从局部到全局动态扩散和高阶时空语义融合为技术突破点，解决人与环境互动场景的级联语义关系推理、语义特征可辨别的虚拟人体生成、长时一致动态场景生成三个关键科学技术问题。拟实现具有常识语义、时序自然的动态场景生成框架，并在室内外开放场景应用验证，预期使场景图生成准确率从主流的40%提升到65%，动态场景人体生成误差从主流的15%降低到5%以内，动态场景生成误差从主流的25%降低为8%以内，为实现虚拟现实和人工智能技术广泛应用提供可实用的基础算法支撑。

Auto-generation of interactive scenes is critical in the field of virtual reality, which involves scene semantic understanding, human shape generation based on videos, and dynamic virtual scene generation. In this field, deep learning has already got great achievements with the large scale of well-labeled datasets, however, three main challenges hinder the generation of high-quality models: data-hungry trained models, lacking semantic details of the generated human bodies, and poor transferability across scenarios of the interactive sequences. Our project exploits the intrinsic semantic relations in videos, which are of commonsense Knowledge (CSK) in spatial and temporal dimensions. We find that the CSK could drive the auto-generation of the interactive scenes. Specifically, we propose three novel solutions to facilitate the generation quality, including, the bidirectional message passing between CSK in the scene graph, a dynamic diffusion for a human model generation when occluded, and high-order temporal features in multiple ranges. Such that we could make progress in three aspects, including the cascade scene graph generation, semantic distinguished human shape generation, and the long-term consistent dynamic scene generation..Our project is to provide a dynamic virtual scene generation framework with common sense knowledge and to take a step further to apply it to the human-environment interactive scenes in the wild. Compared with the state-of-the-art, the accuracy rates of scene graph generation is to increase from 40% to 65%, and the error of human body generation of dynamic scenes is to decrease from 15% to less than 5%, and the dynamic scene generation is to decrease from 20% to less than 5%. In this way, our project could extend the theory and applications of virtual reality and artificial intelligence technology in the industry.