ERI: Resource-Aware Single-Photon Image Sensors

该奖项全部或部分根据2021年美国救援计划法案（公法117-2）资助。光由称为光子的离散能量包组成。相机通过在每个像素处将光子转换成电信号来捕获图像。传统的相机需要在每个像素中使用数百或数千个光子来产生高质量的图像。单光子雪崩二极管（SPAD）是一种新兴的传感器技术，具有独特的能力，可以检测单个光子，时间分辨率低至皮秒。SPAD是非常敏感的光传感器。这使它们能够在传统相机难以形成图像的挑战性场景（例如高速运动、极端照明、低信号功率和长距离）中捕获场景信息。尽管有这些令人兴奋的功能，SPAD相机面临着一个严重的实际挑战，限制了更广泛的使用-高分辨率SPAD相机产生的原始数据量超过了现有数据传输总线的带宽几个数量级。因此，目前大多数商用SPAD相机只能支持低分辨率成像。该项目将开发高带宽效率的光子数据捕获和处理技术，使未来的百万像素甚至更高分辨率的SPAD相机成为可能。这将对消费者摄影、荧光显微镜、生物医学成像以及机器人和自动驾驶汽车的视觉传感器等众多应用产生影响。该项目将涵盖带宽高效的单光子计算成像算法的理论和实际实现，以实现高空间分辨率SPAD相机，用于被动强度成像和主动飞行时间深度传感。遗憾的是，高分辨率SPAD相机的数据瓶颈挑战无法通过逐步提高标准数据传输总线（如USB、PCIe或CameraLink）的带宽来解决。该提案将通过SPAD像素阵列的硬件可重构性和算法可重构性来解决数据带宽挑战。硬件可重构性的第一个推力将建立一个严格的理论基础，在不同水平的场景照明和像素设计规范下，SPAD相机图像质量和输出数据速率中涉及的各种权衡。算法可重构性的第二个推力将设计简约的、场景自适应的统计和数据驱动的压缩技术，以将光子数据流总结为最小形式。在最小形式中，光子数据将仍然携带高质量强度和深度重建所需的所有信息，但具有几个数量级的低带宽要求。用商用SPAD相机硬件设计的实验原型不仅将提供标准图像质量指标方面的性能评估，而且还将展示在芯片上实施这些算法的实际可行性，例如，作为未来SPAD相机模块中的“光子处理单元”。这项研究的结果将通过会议研讨会、研究生课程和面向高中生的STEM外展活动进行传播。该奖项反映了NSF的法定使命，并被认为值得通过使用基金会的知识价值和更广泛的影响审查标准进行评估来支持。

This award is funded in whole or in part under the American Rescue Plan Act of 2021 (Public Law 117-2). Light is composed of discrete energy packets called photons. A camera captures an image by converting photons into an electrical signal at each pixel. A conventional camera needs hundreds or thousands of photons in each pixel to produce a high quality image. Single-photon avalanche diodes (SPADs) are an emerging sensor technology with the unique ability to detect individual photons with a time resolution down to picoseconds. SPADs are extremely sensitive light sensors. This enables them to capture scene information in challenging scenarios (e.g. high speed motion, extreme illumination, low signal power, and long distances) where conventional cameras struggle to form an image. Despite these exciting capabilities, SPAD cameras face a serious practical challenge that limits wider use --- the amount of raw data generated by high resolution SPAD cameras exceeds the bandwidths of existing data transfer buses by several orders of magnitude. As a result, most commercial SPAD cameras today can only support low resolution imaging. This project will develop bandwidth-efficient photon data capture and processing techniques that will enable megapixel and even higher resolution SPAD cameras in the future. This will have implications for myriad applications including consumer photography, fluorescence microscopy, biomedical imaging, and vision sensors for robotics and autonomous vehicles.This project will span both the theory and practical implementation of bandwidth-efficient single-photon computational imaging algorithms towards realizing high spatial resolution SPAD cameras for both passive intensity imaging and active time-of-flight depth-sensing. Unfortunately, the data-bottleneck challenge with high resolution SPAD cameras cannot be solved through incremental improvements in the bandwidth of standard data transfer buses (such as USB, PCIe, or CameraLink). This proposal will address the data bandwidth challenge through hardware reconfigurability and algorithmic reconfigurability of a SPAD pixel array. The first thrust of hardware reconfiguratiblity will establish a rigorous theoretical foundation for various trade-offs involved in a SPAD camera image quality and output data-rate, under varying levels of scene illumination and pixel design specifications. The second thrust of algorithmic reconfigurability will design parsimonious, scene-adaptive statistical and data-driven compression techniques to summarize photon data streams into a minimal form. In the minimal form, the photon data will still carry all the information needed for high quality intensity and depth reconstructions, but with several orders of magnitude lower bandwidth requirements. Experimental prototypes designed with commercially available SPAD camera hardware will not only provide performance evaluation in terms of standard image quality metrics but also demonstrate practical feasibility of implementing these algorithms on-chip, for example, as a "photon processing unit" in future SPAD camera modules. The findings from this research will be disseminated via a conference workshop, a graduate level course, and STEM outreach activities for high-school students.This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.