CAREER: Beyond Perspective Cameras: Multi-perspective Imaging, Reconstruction, Rendering, and Projection

透视相机捕捉场景中对象的空间关系，就像它们从单个视点看起来一样。相比之下，多视角相机将从多个视角看到的内容组合成单个图像，并提供了一个潜在有利的成像系统来理解所观察场景的结构。然而，大多数现有的计算机视觉和图形算法并不直接适用于多视角相机。该项目的目标是开发一个完整的框架来表征和设计新的多视角相机和显示器，并将这些系统用于计算机视觉和图形应用。在视觉方面，PI探索了两种类型的真实的多视角相机：第一种扩展了现有的折反射镜，第二种使用特殊形状的孔径。为了对这些相机进行分类，应用射线几何分析来识别重要的射线结构并研究它们对图像失真的影响。PI还研究了几种新的3D重建方法，如三维立体匹配，光线曲率分析和曲率从失真，为广泛的多视角相机。在图形方面，PI通过将消费者投影仪与特殊形状的镜子/透镜相结合，探索了实用的多视角显示架构。这样的显示器可以提供前所未有的灵活性方面的纵横比，尺寸，视野等PI还研究实时多视角光栅化和截头体剔除算法，通过修改传统的渲染流水线。该项目通过让本科生和研究生参与具有理论和应用部分的独立项目，并通过特拉华州大学和特拉华州之间的工程预科项目吸引代表性不足的学生，为教育做出贡献。州立大学。

A perspective camera captures the spatial relationships of objects in a scene as they would appear from a single viewpoint. In contrast, a multi-perspective camera combines what is seen from several viewpoints into a single image and provides a potentially advantageous imaging system for understanding the structure of observed scenes. However, most existing computer vision and graphics algorithms are not directly applicable to multi-perspective cameras. The goal of this project is to develop a complete framework to characterize and design new multi-perspective cameras and displays and to use these systems for computer vision and graphics applications.On the vision front, the PI explores two types of real multi-perspective cameras: the first extends existing catadioptric mirrors and the second uses specially-shaped apertures. To categorize these cameras, a ray geometry analysis is applied to identify important ray structures and study their implications on image distortions. The PI also investigates several new 3D reconstruction methods, such as epsilon stereo matching, ray-curvature analysis, and curvature-from-distortions, for a broad class of multi-perspective cameras. On the graphics front, the PI explores practical multi-perspective display architectures by combining a consumer projector with specially-shaped mirrors/lenses. Such displays can offer an unprecedented level of flexibility in terms of aspect ratio, size, field of view, etc. The PI also investigates real-time multi-perspective rasterization and frustum culling algorithms by modifying the traditional rendering pipeline. The project contributes to education by involving undergraduate and graduate students in self-contained projects with both theory and application components, and attracting under-represented students through the pre-engineering program between the University of Delaware and the Delaware State University.