Collaborative Research: OP: Meta-optical Computational Image Sensors

在现代日常生活中，相机是不可或缺的，它们确实可以很好地捕捉到人眼所感知的场景。近30年前，当数码摄影首次被引入时，它成为了一项颠覆性的技术。从那时起，相机经历了戏剧性的微型化。有了这些随时可供消费者使用的相机，专业人士和业余爱好者能够体验到拍摄、查看和分享照片是多么容易。但机器视觉、机器人或物联网中的许多新兴应用都需要更先进(更小、更低功率和智能)的摄像头。预计这些摄像头不仅可以捕捉图像，还可以提供机器必须如何工作的信息，比如在自动导航中。对于这种类型的场景理解或目标检测问题，当前的系统使用笨重的摄像头与计算机或图形处理单元相结合。不幸的是，这些系统中的大多数都会消耗大量的能量，而且通常不会针对特定的任务进行优化。通过硬件和软件的共同设计，该项目旨在创造能够低功耗、低延迟操作和尺寸紧凑的计算机器视觉传感器。由此产生的传感器可以给自主导航和机器视觉领域带来革命性的变化。此外，该项目将改进对本科生和高中生的培训和教育，重点是将妇女和少数群体纳入光学和机器学习的多学科研究。通过PI与致力于汽车、成像和增强现实面罩的工业实验室的积极参与，科学成果将通过研讨会、研讨会、同行评议出版物和会议向更广泛的科学受众传播。在自主交通、智能家居和城市以及物联网应用中，对紧凑、低功耗和无处不在的图像传感器的需求非常大。许多这样的机器视觉应用程序需要一个电子后端来解释捕获的图像，或者需要更多的信息，而不仅仅是通常用相机捕获的二维强度信息。目前解决这些问题的方法是使用高端、笨重的摄像头来捕捉高质量的图像，然后开发计算昂贵且耗电的计算机视觉算法。通过针对特定应用(包括深度传感和直接解决对象分割、检测和分类等更高级别的计算机视觉任务)共同优化光学和计算成像算法，可以大幅降低这些成像系统的尺寸和功耗。本项目旨在研究和开发一种光前端和互补计算后端的联合优化算法。光学元件是通过高效介电元光学实现的，其中每个散射体构成一个设计参数。结合数值模拟、器件制造和光学表征，该项目旨在开发用于优化传感器的元光学的逆向设计框架；扩展设计框架以共同优化元光学和计算算法，而不对中间表示施加令人望而却步的限制，以及制造和表征用于3D成像和物体检测的元光学传感器。该奖项反映了NSF的法定使命，并通过使用基金会的智力优势和更广泛的影响审查标准进行评估，被认为值得支持。

In modern daily life, cameras are indispensable, and they truly serve an excellent purpose to capture a scene as perceived by a human eye. Digital photography became a disruptive technology when it was first introduced almost 30 years ago. From that time, cameras have undergone dramatic miniaturization. With these cameras readily available to consumers, professionals and hobbyists are able to experience how easily a photo can be captured, viewed, and shared. But many emerging applications in machine vision, robotics or internet of things require ever more advanced (smaller, lower power and intelligent) cameras. These cameras are expected not just to capture images, but also to provide information on how a machine must function, like for example in autonomous navigation. For this type of scene-understanding or object-detection problems, current systems employ bulky cameras combined with a computer or graphical processing unit. Unfortunately, most of these systems consume significant amounts of energy, and often are not optimized for specific tasks. By co-designing the hardware and software together, this project aims to create computational machine vision sensors, capable of low-power, low-latency operation and compact in size. The resulting sensors can revolutionize the field of autonomous navigation and machine vision. Furthermore, this project will improve the training and education of undergraduate and high school students, with a strong emphasis on including women and minority communities, in multi-disciplinary research in optics and machine learning. Through the PI’s active involvement with industrial laboratories working on automotive, imaging and augmented reality visors, the scientific results will be disseminated to a wider scientific audience via seminars, workshops, peer-reviewed publications, and conferences. There is a tremendous need for compact, low-power, and ubiquitous image sensors for applications in autonomous transportation, smart homes and cities, and the Internet of Things. Many of these machine vision applications require an electronic back-end to interpret the captured images or need more information than just the two-dimensional intensity information usually captured in cameras. Current approaches for solving these problems employ high-end, bulky cameras to capture high-quality images and then exploit computationally expensive and power-hungry computer vision algorithms. Both the size and power consumption of these imaging systems can be drastically reduced via co-optimizing the optics and computational imaging algorithms for specific applications, including depth sensing and directly solving higher-level computer vision tasks such as object segmentation, detection, and classification. This project aims to research and develop such a co-optimization algorithm for an optical front-end and complementary computational back end. The optical elements are implemented via high-efficiency dielectric meta-optics, where each scatterer constitutes a design parameter. Combining numerical simulation, device fabrication, and optical characterization, this project aims to develop an inverse design framework for optimizing the sensor’s meta-optics; expand the design framework to co-optimize both the meta-optics and computational algorithms without placing prohibitive constraints on intermediate representations, as well as fabricate and characterize the meta-optical sensors for 3D imaging and object detection.This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.