VEC: Small: Collaborative Research: Wide Field of View Monocentric Computational Light Field Imaging

该项目的目标是开发单中心相机系统，用于小型设备形状因子中的高分辨率，宽视场（FOV）光场成像。基于最近开发的单中心光学器件的优势-超高分辨率、小物理尺寸、低重量和高光收集-单中心光场成像仪为一系列未来体验式成像和计算应用提供了一个变革性平台。特别地，光场使能单中心光学器件允许用于复杂和宽FOV场景的空间变化的数字聚焦、3D成像能力、立体视图合成和通过部分遮挡器的成像。与任何现有技术相反，单中心光场成像器使得能够实现用于新兴头戴式显示器的沉浸式内容，其支持用低成本的移动的设备捕获焦点提示。一系列计算机视觉算法直接受益于目标计算成像平台，包括4D特征检测、定位和映射、分割、识别、跟踪、深度估计、抠图、物体去除和孔洞填充。所开发的单中心光场成像系统为整个社会带来了好处;在小型设备中提供的3D图像捕获和编辑功能可能会深刻影响未来的人际数字通信，远程协作和教育以及车辆的远程操作。新开发的计算机视觉算法有利于自主车辆的导航。可以轻松录制和编辑各种应用的实时内容，例如用于模拟、培训、恐惧症治疗和文化遗产。光场光学和算法设计将被紧密地整合到斯坦福大学和UCSD的多个研究生课程的教学大纲中，并通过在线学习平台提供给行业专业人士。利用来自加州圣地亚哥大学和斯坦福大学的PI的专业知识，该项目旨在（i）通过单中心光学器件、共形微透镜和光纤耦合设计和制造宽视场光场成像仪，（ii）开发端到端计算成像管道，从编码捕获到新兴的头戴式显示器上的显示，以及（iii）评估计算机视觉和场景理解算法，包括特征检测、定位、映射、分割、分类、跟踪、遮片、分类和对象去除。驱动这个项目的研究问题是寻求一个小型的计算成像系统，该系统足够灵活，可以解锁一系列视觉和体验计算应用程序，这些应用程序目前无法通过相机轻松提供。单中心光学器件为此类应用提供了巨大的优势：宽视场、高分辨率、高聚光度和小外形尺寸。然而，未来的视觉计算应用需要更多的功能：3D成像，大FOV范围内的自适应数字对焦，与新兴的虚拟和增强现实显示器的兼容性，增强的图像编辑模式，如对象分割，删除，插入，定位等。

This project targets the development of monocentric camera systems for high-resolution, wide field-of-view (FOV) light field imaging in small device form factors. Building on the benefits of recently-developed monocentric optics - ultra-high resolution, small physical footprint, low weight, and high light collection - monocentric light field imagers provide a transformative platform for a range of future experiential imaging and computing applications. In particular, light field-enabled monocentric optics allow for spatially-varying digital focus for complex and wide FOV scenes, 3D imaging capabilities, stereo view synthesis, and imaging through partial occluders. As opposed to any existing technology, monocentric light field imagers enable immersive content for emerging head-mounted displays with support for focus cues to be captured with low-cost, mobile devices. A range of computer vision algorithms directly benefit from the targeted computational imaging platform, including 4D feature detection, localization and mapping, segmentation, recognition, tracking, depth estimation, matting, object removal, and hole filling. The developed monocentric light field imaging system provides benefits for society at large; the enabled 3D image capture and editing capabilities offered in a small device form factor could profoundly impact future means of inter-personal digital communication, remote collaboration and education as well as remote operation of vehicles. Newly-developed computer vision algorithms are beneficial for navigation of autonomous vehicles. Live content for a range of applications can be easily recorded and edited, for example for simulation, training, phobia treatment, and cultural heritage. Light field optics and algorithm design will be tightly integrated into the syllabus of multiple graduate-level courses at Stanford and UCSD and made available to industry professionals via online learning platforms.This research investigates a viable solution for these challenges and provides a next-generation computational imaging platform. Leveraging the expertise of PIs from University of California San Diego and Stanford University, this project aims at (i) designing and fabricating a wide field of view light field imager via monocentric optics, conformal microlenses, and fiber coupling, (ii) developing end-to-end computational imaging pipelines, from coded capture to display on emerging head mounted displays, and (iii) evaluating computer vision and scene understanding algorithms, including feature detection, localization, mapping, segmentation, classification, tracking, matting, classification, and object removal. The research question driving this project is the quest for a small, computational imaging system that is flexible enough to unlock a range of visual and experiential computing applications that cannot be easily provided by cameras available today. Monocentric optics offer great benefits for such applications: wide field of view, high resolution, high light collection, and a small form factor. Yet, future visual computing applications require even more functionality: 3D imaging, adaptive digital focus over a large FOV, compatibility with emerging virtual and augmented reality displays, enhanced image editing modes, such as object segmentation, removal, insertion, localization, and more.