Surround 3D video content creation with moving cameras

该项目的总体目标是开发图像采集和处理技术，将真人动作镜头与计算机图形图像相结合，以相对低成本的方式生成3D环绕视频内容。特别是，我们将研究一种新型成像系统的实用性，该系统由一对高分辨率立体相机和一对固定的低分辨率相机组成。成像系统将是移动的，并能够跟踪感兴趣的对象，从而使立体对保持专注于与正在生成的电影内容相关的场景的各个方面。跟踪功能除了允许使用单个立体摄像机对外，还将在摄像机系统平移时使用非跟踪摄像机时产生的运动模糊降至最低。我们将研究图像获取(摄像机跟踪运动)和图形内容生成的脚本编写方法，这些方法将允许内容制作者将观众的注意力引导到与内容叙事相关的场景特征上，同时仍提供环绕沉浸感。该项目还将解决3D环绕内容生成中的一些技术问题。第一个是尽量减少摄像机多角度产生的图像拼接缺陷。这会产生视差，这对于为取景提供深度线索很重要，但在拼接图像以形成全景图时会出现问题。我们建议实现半遮挡感知的技术(半遮挡是一只眼睛看到另一只眼睛看不到的场景细节的地方)来通知缝合过程。我们还将研究CGI全景图和实景(相机)全景图的耦合生成，以最大限度地减少合成步骤中产生的伪像。

The overall goal of the project is to develop image acquisition and processing technology that will enable relatively low-cost generation of 3D surround video content, integrating live action footage with computer graphic imagery. In particular we will investigate the utility of a novel imaging system that consists of a single high resolution stereo pair of cameras combined with a fixed ring of lower resolution cameras. The imaging system will be mobile and able to track objects of interest, thereby keeping the stereo pair fixated on aspects of the scene relevant to the cinematic content being generated. The tracking capability, in addition to permitting a single stereo camera pair to be used, also minimizes motion blur that would otherwise arise with non-tracking cameras as the camera system is translated. We will investigate approaches to scripting of the image acquisition (camera tracking motions) and graphics content generation that will allow the content producer to bias the viewer's attention towards scene features relevant to the content's narrative, while still providing a sense of surround immersion.The project will also address finding solutions to a number of technical problems in the generation of 3D surround content. The first is to minimize image stitching defects arising from the multiple perspectives of the cameras. These generate parallax, which is important to provide depth cues for the viewings, but problematic when stitching images to form panoramas. We propose to implement techniques that are half-occlusion aware (half-occlusions are places where one eye sees scene details that the other eye does not) to inform the stitching process. We will also investigate the coupled generation of the CGI panoramas and the live action (camera) panoramas to minimize artifacts created during the compositing step.