Deep Learning for Free-Viewpoint Video in Sports and Immersive VR Experiences

虚拟现实(VR)和增强现实(AR)体验的日益普及导致了对高质量沉浸式内容的需求。到目前为止，这些体验主要以艺术家制作的内容为特色，但以这种方式创建逼真的模型和纹理的过程具有挑战性和耗时。最近，人们开始使用基于视觉的方法来从真实图像创建虚拟内容，包括自由视点视频(FVV)。FVV允许我们重建真实世界的场景，并将它们呈现在虚拟媒体中。大多数FVV方法需要大量的几何体和纹理数据，并且可能需要大量的处理能力来进行实时渲染。除了这些挑战之外，体育场景对于FVV来说是一个特别困难的主题，因为它的环境不受限制，相当大的表演量和大量的人。本文主要研究体育等高动态场景的FVV离线处理和实时绘制方法。FVV管道使用场景数据的几何表示，并使用相机图像对其进行纹理处理。几何图形通常是通过从多个摄像机视点进行3D重建获得的--无论是常规视点还是基于模型的视点。一般的多视点重建方法不依赖于场景结构的先验信息，许多方法已被应用于FVV内容的生成。基于模型的重建方法包括将形状模型与真实世界的数据进行拟合，以获得场景的几何代理。我们认为，基于模型的方法可以用于产生运动和动态场景的FVV，提供了比现有方法多个好处：-基于模型的表示只需要一组模型参数加上身体模型的开销，而不是每个人和帧的唯一几何形状。这意味着基于模型的FVV将很容易扩展到多人。-由于基于模型的重建的输出随时间本质上是一致的，因此纹理图上的视频压缩更有效，进一步增加了紧凑性。-基于模型的重建对一般重建方法容易发生的错误更稳健。嵌入在模型中的人体形状和姿势的先验知识有助于克服输入数据中存在的视觉模糊，提供没有缺失区域的全身重建。-该模型可以允许在其他捕捉方法不充分的地方推断更精细的细节，例如面部和手周围。-如果身体模型以关节骨骼为特征，则重建可以容易地由艺术家改进或重新激活。基于模型的重建的优点使其适合于FVV，特别是对于运动场景。这项工作的重点是将其应用于VR和AR的沉浸式内容制作。其目的是实现具有时间相干表示的实时FVV渲染，以用于动态场景数据的紧凑流传输。特别关注的是在体育方面的应用，包括球员表现和分析，以及为VR和AR体验创建身临其境的内容。用于体育场景的FVV制作是一个特别具有挑战性的应用，因为环境是不受控制的。相关的困难包括不准确的校准、不受控制的照明、糟糕的分割和非常宽的基线相机。

The increase in popularity of virtual reality (VR) and augmented reality (AR) experiences has given rise to a need for high-quality immersive content. Until now these experiences have predominantly featured artist-made content, but the process of creating realistic models and textures this way is challenging and time-consuming. There has been recent uptake in the use of vision-based methods for creating virtual content from real images, including Free-Viewpoint Video (FVV). FVV allows us to reconstruct real-world scenes and present them in a virtual medium. Most FVV methods require large volumes of geometry and texture data, and can require significant processing capabilities for real-time rendering. On top of these challenges, sports scenes are an especially difficult subject for FVV due to the unconstrained environment, sizeable performance volume andlarge number of people. The overall focus of this work is to investigate methods for the offline processing and real-time rendering of FVV of highly dynamic scenes such as sports. FVV pipelines employ a geometric representation of the scene data, which is textured using camera images. The geometry is usually obtained using 3D reconstruction from multiple camera viewpoints - either general or model-based. General multi-view reconstruction methods are not reliant on prior information about the scene structure, and many have been applied to the generation of FVVcontent. Model-based reconstruction methods involve fitting a shape-model to real-world data to obtain a geometrical proxy of the scene. We believe that model-based methods can be employed in the production of FVV of sports and dynamic scenes, providing multiple benefits over existing methods:- Rather than a unique geometry per person and frame, a model-based representation requires only a set of model parameters plus the overhead of the body model. This means that modelbased FVV would be easily scalable to multiple people.- Since the output of model-based reconstruction is intrinsically consistent over time, video compression on the texture maps is more effective, further increasing compactness.- Model-based reconstruction is more robust to the errors that general reconstruction methods are prone to. Prior knowledge of human shape and pose embedded in the model assists in overcoming visual ambiguities present in the input data, providing a whole-body reconstruction without missing regions.- The model may allow finer detail to be inferred where other capture methods are insufficient, such as around the face and hands.- If the body model features an articulated skeleton, the reconstruction can be easily refined or reanimated by an artist.The merits of model-based reconstruction make it suitable for FVV, especially for sports scenes. This work focuses on employing it in the production of immersive content for VR and AR. The aim is to achieve real-time FVV rendering with temporally coherent representations for compact streaming of dynamic scene data. A particular focus will be on the application to sports, both for player performance and analytics, and the creation of mmersive content for VR and AR experiences. FVV production for sports scenes is an especially challenging application since the environment is uncontrolled. The associated difficulties include inaccurate calibration, uncontrolled illumination, poor segmentation, and very wide-baseline cameras.