VEC: Small: Collaborative Research: Wide Field of View Monocentric Computational Light Field Imaging

该项目旨在开发用于高分辨率、宽视场（FOV）光场成像的单中心相机系统。基于最近开发的单中心光学系统的优势——超高分辨率、小物理占地、低重量和高光采集——单中心光场成像仪为一系列未来的体验式成像和计算应用提供了一个变革性的平台。特别是，光场单心光学系统支持复杂和宽视场场景的空间变化数字焦点、3D成像能力、立体视图合成和通过部分遮挡器成像。与现有的任何技术相比，单心光场成像仪可以为新兴的头戴式显示器提供沉浸式内容，并支持用低成本的移动设备捕获焦点线索。一系列计算机视觉算法直接受益于有针对性的计算成像平台，包括四维特征检测、定位和映射、分割、识别、跟踪、深度估计、抠图、目标去除和补孔。研制的单心光场成像系统为社会带来了广泛的效益；小型设备提供的3D图像捕获和编辑功能将对未来的个人数字通信、远程协作和教育以及车辆远程操作产生深远影响。新开发的计算机视觉算法有利于自动驾驶汽车的导航。一系列应用程序的实时内容可以很容易地录制和编辑，例如模拟，培训，恐惧症治疗和文化遗产。光场光学和算法设计将紧密整合到斯坦福大学和加州大学圣地亚哥分校的多个研究生课程的教学大纲中，并通过在线学习平台提供给行业专业人士。本研究为这些挑战提供了可行的解决方案，并提供了下一代计算成像平台。利用加州大学圣地亚哥分校和斯坦福大学pi的专业知识，该项目旨在(i)通过单心光学，共形微透镜和光纤耦合设计和制造宽视场光场成像仪，（ii）开发端到端计算成像管道，从编码捕获到显示在新兴的头戴式显示器上，以及（iii）评估计算机视觉和场景理解算法，包括特征检测，定位，映射、分割、分类、跟踪、抠图、分类和对象移除。推动该项目的研究问题是寻求一种小型的计算成像系统，该系统足够灵活，可以解锁一系列视觉和体验性计算应用，这些应用是目前可用的相机无法轻易提供的。单心光学为此类应用提供了巨大的好处：宽视场，高分辨率，高光收集和小尺寸。然而，未来的视觉计算应用需要更多的功能：3D成像、大视场上的自适应数字对焦、与新兴的虚拟和增强现实显示器的兼容性、增强的图像编辑模式，如对象分割、移除、插入、定位等。

This project targets the development of monocentric camera systems for high-resolution, wide field-of-view (FOV) light field imaging in small device form factors. Building on the benefits of recently-developed monocentric optics - ultra-high resolution, small physical footprint, low weight, and high light collection - monocentric light field imagers provide a transformative platform for a range of future experiential imaging and computing applications. In particular, light field-enabled monocentric optics allow for spatially-varying digital focus for complex and wide FOV scenes, 3D imaging capabilities, stereo view synthesis, and imaging through partial occluders. As opposed to any existing technology, monocentric light field imagers enable immersive content for emerging head-mounted displays with support for focus cues to be captured with low-cost, mobile devices. A range of computer vision algorithms directly benefit from the targeted computational imaging platform, including 4D feature detection, localization and mapping, segmentation, recognition, tracking, depth estimation, matting, object removal, and hole filling. The developed monocentric light field imaging system provides benefits for society at large; the enabled 3D image capture and editing capabilities offered in a small device form factor could profoundly impact future means of inter-personal digital communication, remote collaboration and education as well as remote operation of vehicles. Newly-developed computer vision algorithms are beneficial for navigation of autonomous vehicles. Live content for a range of applications can be easily recorded and edited, for example for simulation, training, phobia treatment, and cultural heritage. Light field optics and algorithm design will be tightly integrated into the syllabus of multiple graduate-level courses at Stanford and UCSD and made available to industry professionals via online learning platforms.This research investigates a viable solution for these challenges and provides a next-generation computational imaging platform. Leveraging the expertise of PIs from University of California San Diego and Stanford University, this project aims at (i) designing and fabricating a wide field of view light field imager via monocentric optics, conformal microlenses, and fiber coupling, (ii) developing end-to-end computational imaging pipelines, from coded capture to display on emerging head mounted displays, and (iii) evaluating computer vision and scene understanding algorithms, including feature detection, localization, mapping, segmentation, classification, tracking, matting, classification, and object removal. The research question driving this project is the quest for a small, computational imaging system that is flexible enough to unlock a range of visual and experiential computing applications that cannot be easily provided by cameras available today. Monocentric optics offer great benefits for such applications: wide field of view, high resolution, high light collection, and a small form factor. Yet, future visual computing applications require even more functionality: 3D imaging, adaptive digital focus over a large FOV, compatibility with emerging virtual and augmented reality displays, enhanced image editing modes, such as object segmentation, removal, insertion, localization, and more.