CAREER: Multimodal Photodetectors

随着深度学习的最新进展，机器智能获得了前所未有的力量。当与感官功能相结合时，即使是具有基本智能的自动机器也有望彻底改变世界经济。今天，大多数机器的视觉都是基于场景的传统强度图片，就像人类使用的一样。这种视觉方式有许多局限性：它会受到雾和雨的影响，除了三种基本颜色的组合之外，它没有提供光谱信息。由于其严格的安全性和可靠性要求，这些问题极大地限制了自动机器的实际使用。因此，昂贵的光学仪器被用来帮助传统的视觉完成特殊的任务。拟议的项目有可能克服传统成像技术的基本问题。它基于一种新型的光传感像素，可以测量光的多模态信息，如入射角，波长和相位。它们可以提供前所未有的场景感知能力，在未来的机器中广泛使用。强度信息对于常规应用（例如摄影）是足够的，其局限性在高级视觉任务中变得明显。本项目将开发一种新型光电探测器，用于测量光波的多模态信息。它们结构紧凑，可以形成高密度阵列作为成像芯片。虽然多模态信息可以通过传统的光学元件，如透镜，棱镜和光栅测量，这些组件是昂贵的集成。它们还降低了空间分辨率并降低了操作速度。该项目使用新型纳米结构来开发独特的光学相互作用。多模态像素将使用全波模拟设计，并与光刻制造。多模态像素与现有的半导体制造设施完全兼容，并且可能以消费电子产品的成本进行大规模生产。该项目还将开发新的机器学习算法，利用多模态信息执行视觉任务，远远超出当今仅采用强度方法的可能性。该奖项反映了NSF的法定使命，并通过使用基金会的知识价值和更广泛的影响审查标准进行评估，被认为值得支持。

Machine intelligence has acquired unprecedented power with the recent progress of deep learning. When paired with sensory functions, autonomous machines with even rudimentary intelligence are expected to revolutionize the world's economy. Today, the vision that most machines have is based on traditional intensity pictures of a scene, just as humans use. This vision modality has many limitations: it is impaired by fog and rain, and it offers no spectral information other than combinations of three fundamental colors. These issues greatly limit the practical use of autonomous machines due to their stringent safety and reliability requirements. As a result, expensive optical instruments are being used to assist conventional vision in accomplishing special tasks. The proposed project has the potential to overcome the fundamental issues of traditional imaging technologies. It is based on a new type of light-sensing pixels that can measure multimodal information of light, such as incident angle, wavelength, and phase. They could offer unprecedented scene awareness for pervasive use in future machines.Light-sensitive pixels used in today's camera can only detect the intensity of light. The intensity information is sufficient for conventional applications such as photography, its limitations become apparent in advanced vision tasks. This project will develop a new class of photodetectors to measure multimodal information of light waves. They are compact and can form high density arrays as imaging chips. Although multimodal information can be measured through conventional optical components, such as lenses, prisms, and gratings, these components are expensive to integrate. They also degrade spatial resolution and decrease operational speed. This project uses novel nanostructures to exploit unique optical interactions. Multi-modal pixels will be designed using full wave simulation and fabricated with photo-lithography. The multimodal pixels are completely compatible with existing semiconductor fabrication facilities and could potentially be mass-produced at the cost of consumer electronics. The project will also develop new machine learning algorithms to exploit multimodal information to perform vision tasks far beyond those possible with today's intensity-only approach.This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.