ERI: Resource-Aware Single-Photon Image Sensors

点击翻译按钮获取中文摘要

This award is funded in whole or in part under the American Rescue Plan Act of 2021 (Public Law 117-2). Light is composed of discrete energy packets called photons. A camera captures an image by converting photons into an electrical signal at each pixel. A conventional camera needs hundreds or thousands of photons in each pixel to produce a high quality image. Single-photon avalanche diodes (SPADs) are an emerging sensor technology with the unique ability to detect individual photons with a time resolution down to picoseconds. SPADs are extremely sensitive light sensors. This enables them to capture scene information in challenging scenarios (e.g. high speed motion, extreme illumination, low signal power, and long distances) where conventional cameras struggle to form an image. Despite these exciting capabilities, SPAD cameras face a serious practical challenge that limits wider use --- the amount of raw data generated by high resolution SPAD cameras exceeds the bandwidths of existing data transfer buses by several orders of magnitude. As a result, most commercial SPAD cameras today can only support low resolution imaging. This project will develop bandwidth-efficient photon data capture and processing techniques that will enable megapixel and even higher resolution SPAD cameras in the future. This will have implications for myriad applications including consumer photography, fluorescence microscopy, biomedical imaging, and vision sensors for robotics and autonomous vehicles.This project will span both the theory and practical implementation of bandwidth-efficient single-photon computational imaging algorithms towards realizing high spatial resolution SPAD cameras for both passive intensity imaging and active time-of-flight depth-sensing. Unfortunately, the data-bottleneck challenge with high resolution SPAD cameras cannot be solved through incremental improvements in the bandwidth of standard data transfer buses (such as USB, PCIe, or CameraLink). This proposal will address the data bandwidth challenge through hardware reconfigurability and algorithmic reconfigurability of a SPAD pixel array. The first thrust of hardware reconfiguratiblity will establish a rigorous theoretical foundation for various trade-offs involved in a SPAD camera image quality and output data-rate, under varying levels of scene illumination and pixel design specifications. The second thrust of algorithmic reconfigurability will design parsimonious, scene-adaptive statistical and data-driven compression techniques to summarize photon data streams into a minimal form. In the minimal form, the photon data will still carry all the information needed for high quality intensity and depth reconstructions, but with several orders of magnitude lower bandwidth requirements. Experimental prototypes designed with commercially available SPAD camera hardware will not only provide performance evaluation in terms of standard image quality metrics but also demonstrate practical feasibility of implementing these algorithms on-chip, for example, as a "photon processing unit" in future SPAD camera modules. The findings from this research will be disseminated via a conference workshop, a graduate level course, and STEM outreach activities for high-school students.This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.