Efficient Reconstruction and Rendering of Dynamic, Wide-Range Lightfields

基于图像的绘制（IBR）是计算机视觉和图形学领域的一个重要研究课题.从一组先前捕获的图像渲染场景作为场景光场的样本。取决于该方法，不仅彩色图像而且深度图像可以指导新颖视图渲染。以前，IBR的研究主要集中在静态场景，或者使用昂贵的2D摄像机场进行捕获，或者对观察到的场景施加约束，如人物模型，这需要昂贵的全局优化。所要求的研究调查大规模采集，动态表示和考虑动态场景几何的场景时变光场的高效渲染，反射率和照明条件。由于真实的场景在空间和时间上表现出很大程度的相干性，因此我们将研究捕获的不是完整的密集光场，而是光场的稀疏时空部分。时空光场由水平多相机装置捕获，该装置由25个相机组成，总宽度为2.5 m，每个相机获取30 fps。该装置是电机控制的，并且可以在2.5米垂直和水平的2D范围内移动，因此能够随着时间的推移捕获光场的1D切片。同时，通过25个角度样本查看场景点，这允许估计反射率和照明特性。此外，几个深度相机（Kinect/ToF）捕获场景深度。该提案的中心假设是，光场的稀疏采样与深度补偿多视图插值相结合，可以实现密集光场的高质量重建。这种重建使一系列重要的视觉应用，如自由视点电视和自动立体视差显示器的内容生成。为了达到研究目标，我们围绕两个中心方法的建议。首先，我们将调查本地成本函数的有效和强大的优化，积累密集的动态光场从稀疏的数据采集，通过评估场景的几何形状和运动的反射率和照明的考虑。其次，我们将采用一种合适的几何和图像的混合表示来重建和存储动态密集光场。所有方法都朝着高计算效率的方向调整，以便在最短的时间内捕获，存储和渲染新的视图。

Image based rendering (IBR) is an important research theme in both computer vision and graphics. A scene is rendered from a set of previously captures images as samples of the scene light field. Depending on the approach, not only color images but also depth images may guide the novel view rendering. Previously, IBR research has focused mainly on static scenes or used expensive 2D camera fields for capture, or have applied constraints regarding the observed scenes, like models of human characters, which require expensive global optimization.The requested research investigates the large-scale acquisition, the dynamic representation, and the efficient rendering of time-varying light fields of scenes with consideration of dynamic scene geometry, reflectance and illumination conditions. Since real scenes exhibit a large degree of coherence in space and time, we will investigate to capture not the complete dense light field, but to acquire sparse space-time sections of the light field. The space-time light field is captured by a horizontal multi-camera rig, consisting of 25 cameras with total width of 2.5m, acquiring 30 fps per camera. The rig is motor- controlled and shiftable over a 2D range of 2.5m vertical and horizontal, hence capable of capturing 1D-slices of the light field over time. Simultaneously, a scene point is viewed by 25 angular samples which allows to estimate reflectance and illumination properties. In addition, several depth cameras (Kinect/ToF) capture scene depth.The central assumption of this proposal is that sparse sampling of the light field in combination with depth-compensated multi-view interpolation allows for high-quality reconstruction of the dense light field. This reconstruction enables a series of important visual applications, like content-generation for free-viewpoint television and auto-stereoscopic parallax displays.To reach the research goals, we have centered the proposal around two central approaches. Firstly we will investigate local cost functions for efficient and robust optimisation to accumulate the dense dynamic light field from a sparse data acquisition, by evaluating scene geometry and motion under consideration of reflectivity and illumination. Secondly, we will employ a suitable hybrid representation of geometry and imagery for reconstructing and storing the dynamic dense light field. All approaches are tuned towards high computational efficiency in order to capture, store, and render novel views in minimal time.