CAREER: The Light Portal: 3D Reconstruction and Visualization Over Space and Time

这个项目的重点是与从图像或视频流中重建和可视化三维物体相关的问题。PI的最终目标是首先开发基础算法，然后实现一个端到端的光门户，可以在不同的时间或空间中捕获和“重新制定”现实世界的物体或事件。光传送门使用许多具有不同视点的摄像机捕捉场景。从这些输入图像中，PI将开发的智能软件将重建一个连贯的参数场景模型，该模型不仅包括几何形状，还包括照明和表面反射率属性（适合存储、编辑和随着时间的推移播放），或者直接插入要显示的完整彩色射线束（适合在空间上实时传输）。然后，这些结果将在一个独特的光场显示器上可视化，该显示器利用一系列投影仪和微透镜屏幕发射大量可控光线，这样任何数量的观众都可以同时感知到视觉依赖的立体效果，就像他们在真实场景中一样。光门户的实现在视觉、图形、图像处理、网络、人机交互、光学等方面提出了许多科学和工程挑战，PI希望能够通过建立他之前的工作并专注于3D重建和可视化中的核心科学问题来成功解决这些挑战。在重建方面，将探索两条道路。一个目标是允许鲁棒3D重建更广泛的场景类型，通过寻找从复杂的几何，照明和表面反射率属性之间的相互作用的不变量。或者，将开发一个反公式来直接估计最终图像；通过纳入难以在几何设置中采用的新约束，例如自然图像统计，即使在明确重建三维模型是病态的情况下，这种反向公式也应该允许更真实的图像合成。这两种方法是互补的，一种针对存储和编辑进行了优化，另一种针对直接传输和可视化进行了优化。在可视化方面，拟议的光场显示器将采用独特的设计，通过新颖的校准和渲染算法实现常规2D显示器和3D屏幕的分离。因此，除了具有自动立体功能外，该显示器还将在分辨率、尺寸、视野等方面提供前所未有的灵活性和可扩展性。更广泛的影响：与目前使用的无处不在的数码相机和视频显示器类似，PI希望光门户能够提供最终的三维记录和观看体验，这可能会导致人们交流、教育和娱乐方式的深刻变化。特别是，3D功能可以为外科培训和远程协作提供新的范例。这个项目也有一个完整的教育组成部分；除了通常的课程开发、学生参与和通过各种渠道（出版物、演示、项目网站等）的广泛传播之外，PI还将通过积极参与肯塔基大学和波多黎各大学之间的交流计划，努力吸引少数民族学生。

The focus of this project is on the problems associated with reconstruction and visualization of three-dimensional objects from images or video streams. The PI's ultimate goal is to first develop the fundamental algorithms underlying, and then to implement, an end-to-end light portal that can capture and "re-enact" real-world objects or events in a different time or space. A light portal captures a scene using a number of cameras with different viewpoints. From these input images, intelligent software to be developed by the PI will either reconstruct a coherent parametric scene model that includes not only geometry but also lighting and surface reflectance properties (suitable for storage, editing, and playback over time), or directly interpolate the complete bundle of color rays to be displayed (suitable for live transmission over space). These results will then be visualized on a unique light-field display that utilizes an array of projectors and micro-lens screens to emit a large collection of controllable light rays, so that any number of viewers can simultaneously perceive view-dependent stereoscopic effects as if they were in the real scene.Realization of the light portal poses many scientific and engineering challenges in vision, graphics, image processing, networking, human-computer interaction, optics, etc., which the PI expects to be able to successfully tackle by building on his prior work and focusing on the core scientific problems in 3D reconstruction and visualization. On the reconstruction front, two paths will be explored. One is aimed at allowing robust 3D reconstruction for a much broader class of scene types by finding invariants from the complex interactions among geometry, lighting and surface reflectance properties. Alternatively, an inverse formulation to estimate directly the final image will be developed; by incorporating new constraints that are difficult to adopt in a geometric setting, such as nature image statistics, this inverse formulation should allow more realistic image synthesis even when explicitly reconstructing the 3D model is ill-conditioned. These two approaches are complimentary to each other, with one optimized for storage and editing, and the other for direct transmission and visualization. On the visualization front, the proposed light-field display will have a unique design that de-couples regular 2D displays and 3D screens, made possible by novel calibration and rendering algorithms. Therefore, in addition to its auto-stereoscopic capability, this display will offer an unprecedented level of flexibility and scalability in terms of resolution, size, field of view, etc.Broader Impacts: In a manner analogous to the ubiquitous digital cameras and video displays in use today, the PI expects the light portal to provide the ultimate three-dimensional recording and viewing experience that may lead to profound changes in the way people communicate, educate, and entertain. In particular, the 3D capability may enable new paradigms for surgical training and tele-collaboration. This project also has an integral educational component; in addition to the usual course development, student participation, and broad dissemination through various venues (publications, demonstrations, a project web site, etc.), the PI will make efforts to attract minority students by taking an active part in the exchange program between the University of Kentucky and the University of Puerto Rico.