CAREER: The Light Portal: 3D Reconstruction and Visualization Over Space and Time

该项目的重点是与从图像或视频流中重建和可视化三维对象相关的问题。PI的最终目标是首先开发基础的基本算法，然后实现端到端的光门户，该门户可以在不同的时间或空间捕获和重新演绎现实世界的对象或事件。光入口使用多个具有不同视点的摄影机捕捉场景。根据这些输入图像，PI开发的智能软件将重建一个连贯的参数场景模型，该模型不仅包括几何图形，还包括照明和表面反射属性(适合随着时间的推移存储、编辑和回放)，或者直接内插要显示的完整彩色射线束(适合在空间上实时传输)。然后，这些结果将在一个独特的光场显示器上可视化，该光场显示器利用一组投影仪和微透镜屏幕发射大量可控的光线，因此任何数量的观众都可以同时感知依赖于视图的立体效果，就像他们在真实场景中一样。光门的实现在视觉、图形、图像处理、网络、人机交互、光学等方面提出了许多科学和工程挑战，PI希望能够通过建立在他以前的工作基础上，并专注于3D重建和可视化的核心科学问题来成功解决这些挑战。在重建方面，将探索两条道路。一个目标是通过从几何、光照和表面反射特性之间的复杂相互作用中寻找不变量，允许对更广泛的场景类型进行稳健的3D重建。或者，将开发直接估计最终图像的逆公式；通过加入在几何设置中难以采用的新约束，例如自然图像统计，该逆公式应该允许更真实的图像合成，即使在显式重建3D模型是病态的情况下也是如此。这两种方法相辅相成，一种针对存储和编辑进行了优化，另一种针对直接传输和可视化进行了优化。在可视化方面，拟议的光场显示器将具有独特的设计，通过新的校准和渲染算法实现常规2D显示器和3D屏幕的分离。因此，除了自动立体功能外，这种显示器在分辨率、大小、视场等方面将提供前所未有的灵活性和可扩展性。Broader影响：以类似于当今使用的无处不在的数码相机和视频显示器的方式，PI期望光门能够提供最终的三维记录和观看体验，这可能会导致人们交流、教育和娱乐方式的深刻变化。具体地说，3D能力可以实现外科培训和远程协作的新范例。该项目也有一个完整的教育部分；除了通常的课程开发、学生参与和通过各种场所(出版物、演示、项目网站等)广泛传播外，PI将通过积极参与肯塔基大学和波多黎各大学之间的交流计划，努力吸引少数族裔学生。

The focus of this project is on the problems associated with reconstruction and visualization of three-dimensional objects from images or video streams. The PI's ultimate goal is to first develop the fundamental algorithms underlying, and then to implement, an end-to-end light portal that can capture and "re-enact" real-world objects or events in a different time or space. A light portal captures a scene using a number of cameras with different viewpoints. From these input images, intelligent software to be developed by the PI will either reconstruct a coherent parametric scene model that includes not only geometry but also lighting and surface reflectance properties (suitable for storage, editing, and playback over time), or directly interpolate the complete bundle of color rays to be displayed (suitable for live transmission over space). These results will then be visualized on a unique light-field display that utilizes an array of projectors and micro-lens screens to emit a large collection of controllable light rays, so that any number of viewers can simultaneously perceive view-dependent stereoscopic effects as if they were in the real scene.Realization of the light portal poses many scientific and engineering challenges in vision, graphics, image processing, networking, human-computer interaction, optics, etc., which the PI expects to be able to successfully tackle by building on his prior work and focusing on the core scientific problems in 3D reconstruction and visualization. On the reconstruction front, two paths will be explored. One is aimed at allowing robust 3D reconstruction for a much broader class of scene types by finding invariants from the complex interactions among geometry, lighting and surface reflectance properties. Alternatively, an inverse formulation to estimate directly the final image will be developed; by incorporating new constraints that are difficult to adopt in a geometric setting, such as nature image statistics, this inverse formulation should allow more realistic image synthesis even when explicitly reconstructing the 3D model is ill-conditioned. These two approaches are complimentary to each other, with one optimized for storage and editing, and the other for direct transmission and visualization. On the visualization front, the proposed light-field display will have a unique design that de-couples regular 2D displays and 3D screens, made possible by novel calibration and rendering algorithms. Therefore, in addition to its auto-stereoscopic capability, this display will offer an unprecedented level of flexibility and scalability in terms of resolution, size, field of view, etc.Broader Impacts: In a manner analogous to the ubiquitous digital cameras and video displays in use today, the PI expects the light portal to provide the ultimate three-dimensional recording and viewing experience that may lead to profound changes in the way people communicate, educate, and entertain. In particular, the 3D capability may enable new paradigms for surgical training and tele-collaboration. This project also has an integral educational component; in addition to the usual course development, student participation, and broad dissemination through various venues (publications, demonstrations, a project web site, etc.), the PI will make efforts to attract minority students by taking an active part in the exchange program between the University of Kentucky and the University of Puerto Rico.