Collaborative Research: OP: Meta-optical Computational Image Sensors

在现代日常生活中，相机是不可或缺的，它们真正起到了捕捉人眼所感知的场景的出色作用。数码摄影在近30年前首次问世时就成为了一项颠覆性的技术。从那时起，相机经历了戏剧性的小型化。这些相机随时可供消费者使用，专业人士和业余爱好者能够体验到如何轻松地拍摄，查看和分享照片。但机器视觉、机器人或物联网领域的许多新兴应用都需要更先进（更小、更低功耗、更智能）的摄像头。这些相机不仅能捕捉图像，还能提供机器如何运作的信息，比如自动导航。对于这种类型的场景理解或目标检测问题，当前的系统使用庞大的相机与计算机或图形处理单元相结合。不幸的是，大多数这些系统消耗大量的能量，并且通常没有针对特定的任务进行优化。通过共同设计硬件和软件，该项目旨在创建具有低功耗，低延迟操作和紧凑尺寸的计算机器视觉传感器。由此产生的传感器可以彻底改变自主导航和机器视觉领域。此外，该项目将改善对本科生和高中生的培训和教育，重点是在光学和机器学习的多学科研究中纳入女性和少数民族社区。通过PI积极参与汽车、成像和增强现实护目镜的工业实验室，科学成果将通过研讨会、讲习班、同行评审出版物和会议传播给更广泛的科学受众。在自动驾驶交通、智能家居和城市以及物联网等领域，对紧凑、低功耗和无处不在的图像传感器的需求非常大。许多这些机器视觉应用程序需要一个电子后端来解释捕获的图像，或者需要比通常在相机中捕获的二维强度信息更多的信息。目前解决这些问题的方法是使用高端、笨重的相机来捕捉高质量的图像，然后利用计算成本高、耗电量大的计算机视觉算法。通过共同优化特定应用的光学和计算成像算法，这些成像系统的尺寸和功耗都可以大大降低，包括深度传感和直接解决更高级别的计算机视觉任务，如物体分割、检测和分类。本项目旨在研究和开发这样一种光前端和互补计算后端的协同优化算法。光学元件通过高效介电元光学实现，其中每个散射体构成一个设计参数。结合数值模拟、器件制造和光学表征，该项目旨在开发优化传感器元光学的逆设计框架；扩展设计框架，共同优化元光学和计算算法，而不限制中间表示，以及制造和表征用于3D成像和目标检测的元光学传感器。该奖项反映了美国国家科学基金会的法定使命，并通过使用基金会的知识价值和更广泛的影响审查标准进行评估，被认为值得支持。

In modern daily life, cameras are indispensable, and they truly serve an excellent purpose to capture a scene as perceived by a human eye. Digital photography became a disruptive technology when it was first introduced almost 30 years ago. From that time, cameras have undergone dramatic miniaturization. With these cameras readily available to consumers, professionals and hobbyists are able to experience how easily a photo can be captured, viewed, and shared. But many emerging applications in machine vision, robotics or internet of things require ever more advanced (smaller, lower power and intelligent) cameras. These cameras are expected not just to capture images, but also to provide information on how a machine must function, like for example in autonomous navigation. For this type of scene-understanding or object-detection problems, current systems employ bulky cameras combined with a computer or graphical processing unit. Unfortunately, most of these systems consume significant amounts of energy, and often are not optimized for specific tasks. By co-designing the hardware and software together, this project aims to create computational machine vision sensors, capable of low-power, low-latency operation and compact in size. The resulting sensors can revolutionize the field of autonomous navigation and machine vision. Furthermore, this project will improve the training and education of undergraduate and high school students, with a strong emphasis on including women and minority communities, in multi-disciplinary research in optics and machine learning. Through the PI’s active involvement with industrial laboratories working on automotive, imaging and augmented reality visors, the scientific results will be disseminated to a wider scientific audience via seminars, workshops, peer-reviewed publications, and conferences. There is a tremendous need for compact, low-power, and ubiquitous image sensors for applications in autonomous transportation, smart homes and cities, and the Internet of Things. Many of these machine vision applications require an electronic back-end to interpret the captured images or need more information than just the two-dimensional intensity information usually captured in cameras. Current approaches for solving these problems employ high-end, bulky cameras to capture high-quality images and then exploit computationally expensive and power-hungry computer vision algorithms. Both the size and power consumption of these imaging systems can be drastically reduced via co-optimizing the optics and computational imaging algorithms for specific applications, including depth sensing and directly solving higher-level computer vision tasks such as object segmentation, detection, and classification. This project aims to research and develop such a co-optimization algorithm for an optical front-end and complementary computational back end. The optical elements are implemented via high-efficiency dielectric meta-optics, where each scatterer constitutes a design parameter. Combining numerical simulation, device fabrication, and optical characterization, this project aims to develop an inverse design framework for optimizing the sensor’s meta-optics; expand the design framework to co-optimize both the meta-optics and computational algorithms without placing prohibitive constraints on intermediate representations, as well as fabricate and characterize the meta-optical sensors for 3D imaging and object detection.This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.