Collaborative Research: OP: Meta-optical Computational Image Sensors

在现代日常生活中，相机是必不可少的，它们真正起到了很好的作用，以捕捉人眼所感知的场景。数码摄影在近30年前首次推出时就成为了一项颠覆性的技术。从那时起，相机经历了戏剧性的小型化。有了这些相机，消费者、专业人士和业余爱好者都能体验到拍摄、查看和分享照片是多么容易。但是，机器视觉、机器人或物联网领域的许多新兴应用需要更先进（更小、更低功耗和更智能）的相机。这些摄像头不仅可以捕捉图像，还可以提供机器必须如何运行的信息，例如在自主导航中。对于这种类型的场景理解或对象检测问题，当前的系统采用与计算机或图形处理单元相结合的笨重的相机。不幸的是，这些系统中的大多数消耗大量的能量，并且通常没有针对特定任务进行优化。通过共同设计硬件和软件，该项目旨在创建计算机器视觉传感器，能够低功耗，低延迟操作和紧凑的尺寸。由此产生的传感器可以彻底改变自主导航和机器视觉领域。此外，该项目将改善本科生和高中生的培训和教育，特别强调将妇女和少数民族社区纳入光学和机器学习的多学科研究。通过PI积极参与汽车，成像和增强现实遮阳板的工业实验室，科学成果将通过研讨会，研讨会，同行评审的出版物和会议传播给更广泛的科学受众。 对于紧凑、低功耗和无处不在的图像传感器，在自动交通、智能家居和城市以及物联网中的应用有着巨大的需求。许多机器视觉应用需要电子后端来解释捕获的图像，或者需要比通常在相机中捕获的二维强度信息更多的信息。目前解决这些问题的方法采用高端，笨重的相机来捕获高质量的图像，然后利用计算昂贵和耗电的计算机视觉算法。这些成像系统的尺寸和功耗都可以通过针对特定应用（包括深度传感和直接解决更高级别的计算机视觉任务，如对象分割、检测和分类）共同优化光学和计算成像算法来大幅降低。本项目旨在研究和开发这样一种用于光学前端和互补计算后端的协同优化算法。光学元件通过高效介电元光学器件实现，其中每个散射体构成设计参数。结合数值模拟、器件制作和光学特性，本项目旨在开发一个优化传感器元光学的逆向设计框架;扩展设计框架以共同优化元光学和计算算法，而不对中间表示施加禁止性约束，以及制造和表征Meta-用于3D成像和物体检测的光学传感器。该奖项反映了NSF的法定使命，并通过评估被认为值得支持使用基金会的知识价值和更广泛的影响审查标准。

In modern daily life, cameras are indispensable, and they truly serve an excellent purpose to capture a scene as perceived by a human eye. Digital photography became a disruptive technology when it was first introduced almost 30 years ago. From that time, cameras have undergone dramatic miniaturization. With these cameras readily available to consumers, professionals and hobbyists are able to experience how easily a photo can be captured, viewed, and shared. But many emerging applications in machine vision, robotics or internet of things require ever more advanced (smaller, lower power and intelligent) cameras. These cameras are expected not just to capture images, but also to provide information on how a machine must function, like for example in autonomous navigation. For this type of scene-understanding or object-detection problems, current systems employ bulky cameras combined with a computer or graphical processing unit. Unfortunately, most of these systems consume significant amounts of energy, and often are not optimized for specific tasks. By co-designing the hardware and software together, this project aims to create computational machine vision sensors, capable of low-power, low-latency operation and compact in size. The resulting sensors can revolutionize the field of autonomous navigation and machine vision. Furthermore, this project will improve the training and education of undergraduate and high school students, with a strong emphasis on including women and minority communities, in multi-disciplinary research in optics and machine learning. Through the PI’s active involvement with industrial laboratories working on automotive, imaging and augmented reality visors, the scientific results will be disseminated to a wider scientific audience via seminars, workshops, peer-reviewed publications, and conferences. There is a tremendous need for compact, low-power, and ubiquitous image sensors for applications in autonomous transportation, smart homes and cities, and the Internet of Things. Many of these machine vision applications require an electronic back-end to interpret the captured images or need more information than just the two-dimensional intensity information usually captured in cameras. Current approaches for solving these problems employ high-end, bulky cameras to capture high-quality images and then exploit computationally expensive and power-hungry computer vision algorithms. Both the size and power consumption of these imaging systems can be drastically reduced via co-optimizing the optics and computational imaging algorithms for specific applications, including depth sensing and directly solving higher-level computer vision tasks such as object segmentation, detection, and classification. This project aims to research and develop such a co-optimization algorithm for an optical front-end and complementary computational back end. The optical elements are implemented via high-efficiency dielectric meta-optics, where each scatterer constitutes a design parameter. Combining numerical simulation, device fabrication, and optical characterization, this project aims to develop an inverse design framework for optimizing the sensor’s meta-optics; expand the design framework to co-optimize both the meta-optics and computational algorithms without placing prohibitive constraints on intermediate representations, as well as fabricate and characterize the meta-optical sensors for 3D imaging and object detection.This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.