权益分类	功能权益	普通用户	{{item.name}}会员
{{category.name}}	{{benefitItem.name}}

Deep Learning for Free-Viewpoint Video in Sports and Immersive VR Experiences

体育和沉浸式 VR 体验中自由视点视频的深度学习

基本信息

批准号：
1976240
负责人：
金额：
--
依托单位：
University of Surrey
依托单位国家：
英国
项目类别：
Studentship
财政年份：
2017
资助国家：
英国
起止时间：
2017 至无数据
项目状态：
已结题

来源：
https://gtr.ukri.org/projects?ref=studentship-1976240
关键词：
Deep Learning Free Viewpoint Video

项目摘要

The increase in popularity of virtual reality (VR) and augmented reality (AR) experiences has given rise to a need for high-quality immersive content. Until now these experiences have predominantly featured artist-made content, but the process of creating realistic models and textures this way is challenging and time-consuming. There has been recent uptake in the use of vision-based methods for creating virtual content from real images, including Free-Viewpoint Video (FVV). FVV allows us to reconstruct real-world scenes and present them in a virtual medium. Most FVV methods require large volumes of geometry and texture data, and can require significant processing capabilities for real-time rendering. On top of these challenges, sports scenes are an especially difficult subject for FVV due to the unconstrained environment, sizeable performance volume andlarge number of people. The overall focus of this work is to investigate methods for the offline processing and real-time rendering of FVV of highly dynamic scenes such as sports. FVV pipelines employ a geometric representation of the scene data, which is textured using camera images. The geometry is usually obtained using 3D reconstruction from multiple camera viewpoints - either general or model-based. General multi-view reconstruction methods are not reliant on prior information about the scene structure, and many have been applied to the generation of FVVcontent. Model-based reconstruction methods involve fitting a shape-model to real-world data to obtain a geometrical proxy of the scene. We believe that model-based methods can be employed in the production of FVV of sports and dynamic scenes, providing multiple benefits over existing methods:- Rather than a unique geometry per person and frame, a model-based representation requires only a set of model parameters plus the overhead of the body model. This means that modelbased FVV would be easily scalable to multiple people.- Since the output of model-based reconstruction is intrinsically consistent over time, video compression on the texture maps is more effective, further increasing compactness.- Model-based reconstruction is more robust to the errors that general reconstruction methods are prone to. Prior knowledge of human shape and pose embedded in the model assists in overcoming visual ambiguities present in the input data, providing a whole-body reconstruction without missing regions.- The model may allow finer detail to be inferred where other capture methods are insufficient, such as around the face and hands.- If the body model features an articulated skeleton, the reconstruction can be easily refined or reanimated by an artist.The merits of model-based reconstruction make it suitable for FVV, especially for sports scenes. This work focuses on employing it in the production of immersive content for VR and AR. The aim is to achieve real-time FVV rendering with temporally coherent representations for compact streaming of dynamic scene data. A particular focus will be on the application to sports, both for player performance and analytics, and the creation of mmersive content for VR and AR experiences. FVV production for sports scenes is an especially challenging application since the environment is uncontrolled. The associated difficulties include inaccurate calibration, uncontrolled illumination, poor segmentation, and very wide-baseline cameras.

虚拟现实 (VR) 和增强现实 (AR) 体验的日益普及引发了对高质量沉浸式内容的需求。到目前为止，这些体验主要以艺术家制作的内容为特色，但以这种方式创建逼真的模型和纹理的过程充满挑战且耗时。最近人们开始使用基于视觉的方法从真实图像创建虚拟内容，包括自由视点视频 (FVV)。 FVV 允许我们重建现实世界场景并将其呈现在虚拟介质中。大多数 FVV 方法需要大量几何和纹理数据，并且可能需要强大的实时渲染处理能力。除了这些挑战之外，体育场景由于环境不受限制、演出量大、人数众多，对于FVV来说是一个特别困难的题材。这项工作的总体重点是研究运动等高动态场景的 FVV 离线处理和实时渲染方法。 FVV 管道采用场景数据的几何表示，并使用相机图像进行纹理化。几何形状通常是通过从多个摄像机视点（一般视点或基于模型的视点）进行 3D 重建来获得的。一般的多视图重建方法不依赖于场景结构的先验信息，并且许多方法已应用于 FVV 内容的生成。基于模型的重建方法涉及将形状模型拟合到现实世界数据以获得场景的几何代理。我们相信，基于模型的方法可以用于制作运动和动态场景的 FVV，与现有方法相比具有多种优势：- 基于模型的表示只需要一组模型参数加上身体模型的开销，而不是每个人和帧都有独特的几何形状。这意味着基于模型的 FVV 可以轻松扩展到多人。- 由于基于模型的重建的输出随着时间的推移本质上是一致的，因此纹理图上的视频压缩更加有效，进一步提高了紧凑性。- 基于模型的重建对于一般重建方法容易出现的错误更加鲁棒。模型中嵌入的人体形状和姿势的先验知识有助于克服输入数据中存在的视觉模糊性，提供不丢失区域的全身重建。 - 在其他捕获方法不足的地方，例如面部和手部，该模型可以推断出更精细的细节。 - 如果身体模型具有铰接骨架，则艺术家可以轻松地改进或重新激活重建。基于模型的优点重构使其适合FVV，特别是运动场景。这项工作的重点是将其应用于 VR 和 AR 沉浸式内容的制作中。目的是通过时间相干表示实现实时 FVV 渲染，以实现动态场景数据的紧凑流。特别关注的是体育应用，包括运动员表现和分析，以及为 VR 和 AR 体验创建沉浸式内容。体育场景的 FVV 制作是一项特别具有挑战性的应用，因为环境不受控制。相关的困难包括校准不准确、照明不受控制、分割效果差以及相机基线非常宽。