Efficient Reconstruction and Rendering of Dynamic, Wide-Range Lightfields

基于图像的绘制(IBR)是计算机视觉和图形学领域的一个重要研究课题。从一组先前捕获的图像中渲染场景作为场景光场的样本。根据方法的不同，不仅彩色图像，而且深度图像可以指导新颖的视图绘制。以前，IBR的研究主要集中在静态场景或使用昂贵的2D相机场来捕捉，或者对观察到的场景施加约束，例如需要昂贵的全局优化的人物模型。所要求的研究包括考虑动态场景几何形状、反射率和光照条件的场景的大规模获取、动态表示和场景时变光场的高效绘制。由于真实场景在空间和时间上表现出很大程度的相干性，我们将研究如何捕捉完整的密集光场，而是获取光场的稀疏时空部分。时空光场由一个水平多摄像机装置拍摄，该装置由25个摄像机组成，总宽度为2.5米，每个摄像机采集30帧/秒。该钻机由电机控制，可在2.5米垂直和水平的2D范围内移动，因此能够随时间捕获光场的一维切片。同时，通过25个角度样本来查看场景点，这允许估计反射率和照明属性。此外，多个深度相机(Kinect/ToF)捕捉场景深度。该方案的中心假设是，光场的稀疏采样与深度补偿的多视角内插相结合，可以高质量地重建密集光场。这种重建实现了一系列重要的视觉应用，如自由视点电视的内容生成和自动立体视差显示。为了达到研究目标，我们将该提案集中在两个主要方法上。首先，我们将研究局部代价函数，通过在考虑反射率和光照的情况下评估场景几何和运动，来高效和稳健地优化以积累来自稀疏数据采集的密集动态光场。其次，我们将采用合适的几何和图像的混合表示来重建和存储动态密集光场。所有方法都朝着高计算效率的方向调整，以便在最短的时间内捕获、存储和呈现新的视图。

Image based rendering (IBR) is an important research theme in both computer vision and graphics. A scene is rendered from a set of previously captures images as samples of the scene light field. Depending on the approach, not only color images but also depth images may guide the novel view rendering. Previously, IBR research has focused mainly on static scenes or used expensive 2D camera fields for capture, or have applied constraints regarding the observed scenes, like models of human characters, which require expensive global optimization.The requested research investigates the large-scale acquisition, the dynamic representation, and the efficient rendering of time-varying light fields of scenes with consideration of dynamic scene geometry, reflectance and illumination conditions. Since real scenes exhibit a large degree of coherence in space and time, we will investigate to capture not the complete dense light field, but to acquire sparse space-time sections of the light field. The space-time light field is captured by a horizontal multi-camera rig, consisting of 25 cameras with total width of 2.5m, acquiring 30 fps per camera. The rig is motor- controlled and shiftable over a 2D range of 2.5m vertical and horizontal, hence capable of capturing 1D-slices of the light field over time. Simultaneously, a scene point is viewed by 25 angular samples which allows to estimate reflectance and illumination properties. In addition, several depth cameras (Kinect/ToF) capture scene depth.The central assumption of this proposal is that sparse sampling of the light field in combination with depth-compensated multi-view interpolation allows for high-quality reconstruction of the dense light field. This reconstruction enables a series of important visual applications, like content-generation for free-viewpoint television and auto-stereoscopic parallax displays.To reach the research goals, we have centered the proposal around two central approaches. Firstly we will investigate local cost functions for efficient and robust optimisation to accumulate the dense dynamic light field from a sparse data acquisition, by evaluating scene geometry and motion under consideration of reflectivity and illumination. Secondly, we will employ a suitable hybrid representation of geometry and imagery for reconstructing and storing the dynamic dense light field. All approaches are tuned towards high computational efficiency in order to capture, store, and render novel views in minimal time.