VEC: Small: Collaborative Research: Wide Field of View Monocentric Computational Light Field Imaging

该项目的目标是开发用于小设备外形尺寸的高分辨率、大视场(FOV)光场成像的单中心相机系统。基于最近开发的单中心光学的优点-超高分辨率、小物理占地、低重量和高光收集-单中心光场成像器为未来的一系列体验式成像和计算应用提供了一个变革性的平台。特别是，支持光场的单心光学允许复杂和宽视野场景的空间变化的数字对焦、3D成像能力、立体视图合成以及通过部分遮光器进行成像。与任何现有技术不同，单中心光场成像器能够为新兴的头盔显示器提供身临其境的内容，并支持使用低成本的移动设备捕获焦点提示。一系列计算机视觉算法直接受益于目标计算成像平台，包括4D特征检测、定位和映射、分割、识别、跟踪、深度估计、遮片、对象移除和孔洞填充。开发的单中心光场成像系统为整个社会带来了好处；在小巧的设备外形中提供的启用的3D图像捕获和编辑能力可能会对未来的人与人之间的数字通信、远程协作和教育以及车辆的远程操作产生深远的影响。新发展的计算机视觉算法有利于自主车辆的导航。可以轻松录制和编辑各种应用程序的实况内容，例如用于模拟、培训、恐惧症治疗和文化遗产。光场光学和算法设计将被紧密地整合到斯坦福大学和加州大学圣迭戈分校的多门研究生课程的教学大纲中，并通过在线学习平台向行业专业人士提供。这项研究探索了应对这些挑战的可行解决方案，并提供了下一代计算成像平台。利用加州大学圣地亚哥分校和斯坦福大学PI的专业知识，该项目旨在(I)设计和制造通过单心光学、共形微透镜和光纤耦合的大视场光场成像器，(Ii)开发从编码捕获到在新兴头盔显示器上显示的端到端计算成像管道，以及(Iii)评估计算机视觉和场景理解算法，包括特征检测、定位、地图绘制、分割、分类、跟踪、遮片、分类和对象移除。推动这一项目的研究问题是寻求一种小型的计算成像系统，该系统足够灵活，可以解锁一系列视觉和体验式计算应用程序，而这些应用程序是目前可用的摄像头无法轻松提供的。单心光学为这样的应用提供了巨大的好处：宽视场、高分辨率、高集光率和小形状因数。然而，未来的视觉计算应用需要更多的功能：3D成像，大视场上的自适应数字聚焦，与新兴的虚拟和增强现实显示器的兼容性，增强的图像编辑模式，如对象分割、移除、插入、定位等。

This project targets the development of monocentric camera systems for high-resolution, wide field-of-view (FOV) light field imaging in small device form factors. Building on the benefits of recently-developed monocentric optics - ultra-high resolution, small physical footprint, low weight, and high light collection - monocentric light field imagers provide a transformative platform for a range of future experiential imaging and computing applications. In particular, light field-enabled monocentric optics allow for spatially-varying digital focus for complex and wide FOV scenes, 3D imaging capabilities, stereo view synthesis, and imaging through partial occluders. As opposed to any existing technology, monocentric light field imagers enable immersive content for emerging head-mounted displays with support for focus cues to be captured with low-cost, mobile devices. A range of computer vision algorithms directly benefit from the targeted computational imaging platform, including 4D feature detection, localization and mapping, segmentation, recognition, tracking, depth estimation, matting, object removal, and hole filling. The developed monocentric light field imaging system provides benefits for society at large; the enabled 3D image capture and editing capabilities offered in a small device form factor could profoundly impact future means of inter-personal digital communication, remote collaboration and education as well as remote operation of vehicles. Newly-developed computer vision algorithms are beneficial for navigation of autonomous vehicles. Live content for a range of applications can be easily recorded and edited, for example for simulation, training, phobia treatment, and cultural heritage. Light field optics and algorithm design will be tightly integrated into the syllabus of multiple graduate-level courses at Stanford and UCSD and made available to industry professionals via online learning platforms.This research investigates a viable solution for these challenges and provides a next-generation computational imaging platform. Leveraging the expertise of PIs from University of California San Diego and Stanford University, this project aims at (i) designing and fabricating a wide field of view light field imager via monocentric optics, conformal microlenses, and fiber coupling, (ii) developing end-to-end computational imaging pipelines, from coded capture to display on emerging head mounted displays, and (iii) evaluating computer vision and scene understanding algorithms, including feature detection, localization, mapping, segmentation, classification, tracking, matting, classification, and object removal. The research question driving this project is the quest for a small, computational imaging system that is flexible enough to unlock a range of visual and experiential computing applications that cannot be easily provided by cameras available today. Monocentric optics offer great benefits for such applications: wide field of view, high resolution, high light collection, and a small form factor. Yet, future visual computing applications require even more functionality: 3D imaging, adaptive digital focus over a large FOV, compatibility with emerging virtual and augmented reality displays, enhanced image editing modes, such as object segmentation, removal, insertion, localization, and more.