CAREER: Plenoptic Scene Reconstruction

这是一个为期四年的持续奖励的第一年资助。这个项目的目标是开发一个研究和教育计划，模拟复杂的真实？照片中的世界环境。我们被以下问题所激励：假设一个人可以从任何可能的观看位置和方向拍摄一个场景。通过哪些算法可以推断出场景的结构？为了回答这些问题，我们从全光学函数的角度对场景重建问题进行了正式的研究，全光学函数是一种5D函数，对场景的所有可能图像的空间进行编码。该项目将集中在两个主要问题上：(1)获取真实场景的全视觉表示，(2)从这些表示中重建场景几何和辐射。为了解决第一个问题，我们提出了一种原则性的方法来解决图像采集问题，即，为了获得最佳的场景重建，应该沿着哪个方向对场景亮度进行采样。基于这一分析，我们提出了针对重建任务进行优化的新型全光学相机。第二个问题是设计从全光表示计算场景几何和亮度的算法。与基于透视图像的传统方法不同，本文提出的算法将直接对全视函数进行操作，并整合来自连续视点的信息。由于对应性、可见性和规模，该公式引入了独特的挑战，这些将在提议的工作中解决。提出的工作的一个关键成果将是构建复杂现实世界环境的3D模型的实用技术。这种能力是各种机器人任务的核心，如视觉伺服、机器人导航和运动规划。环境建模也将促进计算机中的许多应用。辅助设计和计算机图形学，包括互联网上远程对象和环境的可视化，电视和电影的虚拟工作室，以及3D虚拟远程会议。该项目的一个组成部分是计算机视觉和图形学场景建模的长期教育计划，强调课程开发、学生研究机会和推广活动。一个重要的组成部分是将视觉、图形和图像处理的概念整合到CMU的本科课程中，并设计新的关于环境捕获和合成的跨学科课程。本科生和研究生的参与是拟议研究计划不可或缺的一部分，学生将被鼓励为正在进行的研究活动做出贡献。外展活动将包括组织教程和课程，向更大的社区传播研究成果。

This is the first year funding of a four year continuing award. The objective of this project is to develop a research and educational program for modeling complex real?world environments from photographs. We are motivated by the following questions: suppose one could photograph a scene from every possible viewing position and orientation. What could be inferred about the structure of the scene and via which algorithms? To answer these questions, we propose a formal study of the scene reconstruction problem from the standpoint of the plenopticfunction, a 5D function that encodes the space of all possible images of a scene.The project will focus on two primary issues: (1) acquiring plenoptic representations of real scenes, and (2) reconstructing scene geo metry and radiance from such representations. To address the first problem, we propose a principled approach to the problem of image acquisition, i.e., along which directions should scene radiance be sampled in order to obtain the best possible scene reconstructions. Based on this analysis, we propose novel plenoptic cameras that are optimized for reconstruction tasks. The second problem is to devise algorithms for computing scene geometry and radiance from plenoptic representations. In contrast to traditional approaches which are based on perspective images, the proposed algorithms will operate directly on the plenoptic function and integrate information from a continuum of viewpoints. This formulation introduces unique challenges due to correspondence, visibility, and scale that will be addressed in the proposed work.A key outcome of the proposed work will be practical techniques for constructing 3D models of complex realworld environments. This capability is central for a variety of robotics tasks such as visual servoing, robot navigation, and motion planning. Environment modeling will also facilitate numerous applications in computer?aided design and computer graphics, including visualization of remote objects and environments over the Internet, virtual studios for television and film, and 3D virtual teleconferencing.An integral part of the project is a long term educational program for scene modeling in computer vision and graphics, emphasizing curriculum development, research opportunity for students, and outreach activities. An important component will be to integrate concepts from vision, graphics, and image processing within the undergraduate curriculum at CMU, and to design new interdisciplinary courses on environment capture and synthesis. The involvement of both undergraduate and graduate students is integral to the proposed research plan, and students will be encouraged to contributed to ongoing research activities. Outreach activities will include organizing tutorials and courses to disseminate research results to the larger community.