Surround 3D video content creation with moving cameras

该项目的总体目标是开发图像采集和处理技术，以相对低成本生成3D环绕视频内容，将实景镜头与计算机图形图像相结合。特别是，我们将研究一种新型成像系统的效用，该系统由单个高分辨率立体相机对与固定环低分辨率相机组合而成。成像系统将是可移动的，能够跟踪感兴趣的物体，从而使立体视觉对固定在与正在生成的电影内容相关的场景方面。跟踪功能，除了允许使用单个立体摄像机对之外，还可以最大限度地减少运动模糊，否则在摄像机系统转换时，非跟踪摄像机会产生运动模糊。我们将研究图像采集（摄像机跟踪运动）和图形内容生成的脚本方法，这些方法将允许内容制作人将观众的注意力偏向于与内容叙述相关的场景特征，同时仍然提供环绕沉浸感。该项目还将致力于寻找3D环绕内容生成中一些技术问题的解决方案。首先是尽量减少由相机的多个角度引起的图像拼接缺陷。这些会产生视差，这对于为观看提供深度线索很重要，但在拼接图像以形成全景时就会出现问题。我们建议实现半遮挡感知技术（半遮挡是一只眼睛看到另一只眼睛看不到的场景细节的地方）来通知拼接过程。我们还将研究CGI全景图和实景（相机）全景图的耦合生成，以尽量减少合成步骤中创建的伪影。

The overall goal of the project is to develop image acquisition and processing technology that will enable relatively low-cost generation of 3D surround video content, integrating live action footage with computer graphic imagery. In particular we will investigate the utility of a novel imaging system that consists of a single high resolution stereo pair of cameras combined with a fixed ring of lower resolution cameras. The imaging system will be mobile and able to track objects of interest, thereby keeping the stereo pair fixated on aspects of the scene relevant to the cinematic content being generated. The tracking capability, in addition to permitting a single stereo camera pair to be used, also minimizes motion blur that would otherwise arise with non-tracking cameras as the camera system is translated. We will investigate approaches to scripting of the image acquisition (camera tracking motions) and graphics content generation that will allow the content producer to bias the viewer's attention towards scene features relevant to the content's narrative, while still providing a sense of surround immersion.The project will also address finding solutions to a number of technical problems in the generation of 3D surround content. The first is to minimize image stitching defects arising from the multiple perspectives of the cameras. These generate parallax, which is important to provide depth cues for the viewings, but problematic when stitching images to form panoramas. We propose to implement techniques that are half-occlusion aware (half-occlusions are places where one eye sees scene details that the other eye does not) to inform the stitching process. We will also investigate the coupled generation of the CGI panoramas and the live action (camera) panoramas to minimize artifacts created during the compositing step.