CAREER: Beyond Perspective Cameras: Multi-perspective Imaging, Reconstruction, Rendering, and Projection

透视相机捕捉场景中对象的空间关系，就像它们从单个视点显示一样。相比之下，多视角相机将从几个视点看到的东西组合成一幅图像，并提供了一种潜在的有利成像系统，用于了解观察到的场景的结构。然而，现有的大多数计算机视觉和图形算法并不直接适用于多视角摄像机。这个项目的目标是开发一个完整的框架来表征和设计新的多视角相机和显示器，并将这些系统用于计算机视觉和图形应用。在视觉方面，PI探索两种类型的真实多视角相机：第一种是扩展现有的折射镜，第二种是使用特殊形状的光圈。为了对这些相机进行分类，应用光线几何分析来识别重要的光线结构，并研究它们对图像失真的影响。PI还研究了几种新的3D重建方法，如epsilon立体匹配、光线曲率分析和曲率失真，适用于广泛的多视角相机。在图形方面，PI通过将消费类投影仪与特殊形状的镜子/透镜相结合，探索了实用的多视角显示架构。这种显示器在纵横比、大小、视场等方面可以提供前所未有的灵活性。PI还通过修改传统的渲染流水线来研究实时多视角光栅化和锥体剔除算法。该项目通过让本科生和研究生参与包含理论和应用部分的独立项目，并通过特拉华大学和特拉华州立大学之间的工程预科项目吸引代表不足的学生，为教育做出了贡献。

A perspective camera captures the spatial relationships of objects in a scene as they would appear from a single viewpoint. In contrast, a multi-perspective camera combines what is seen from several viewpoints into a single image and provides a potentially advantageous imaging system for understanding the structure of observed scenes. However, most existing computer vision and graphics algorithms are not directly applicable to multi-perspective cameras. The goal of this project is to develop a complete framework to characterize and design new multi-perspective cameras and displays and to use these systems for computer vision and graphics applications.On the vision front, the PI explores two types of real multi-perspective cameras: the first extends existing catadioptric mirrors and the second uses specially-shaped apertures. To categorize these cameras, a ray geometry analysis is applied to identify important ray structures and study their implications on image distortions. The PI also investigates several new 3D reconstruction methods, such as epsilon stereo matching, ray-curvature analysis, and curvature-from-distortions, for a broad class of multi-perspective cameras. On the graphics front, the PI explores practical multi-perspective display architectures by combining a consumer projector with specially-shaped mirrors/lenses. Such displays can offer an unprecedented level of flexibility in terms of aspect ratio, size, field of view, etc. The PI also investigates real-time multi-perspective rasterization and frustum culling algorithms by modifying the traditional rendering pipeline. The project contributes to education by involving undergraduate and graduate students in self-contained projects with both theory and application components, and attracting under-represented students through the pre-engineering program between the University of Delaware and the Delaware State University.