Single-Photon Cameras for Extreme Computer Vision and Computational Astronomy

点击翻译按钮获取中文摘要

Cameras capture images by collecting photons: discrete packets of light that arrive to the sensor individually, stochastically, and asynchronously. While this physical process is abstracted away by the vast majority of computer vision algorithms, an emerging class of single-photon cameras provides means to precisely detect and catalog the arrival times of individual photons. Within the past five years, the growing availability of single-photon cameras has led to numerous never-seen-before imaging capabilities, from vision systems that can see in near-total darkness and around corners, to high-precision 3D cameras for automobiles and smartphones, to new biomedical imaging techniques for seeing below the skin. However, the single-photon cameras currently used by various laboratories around the world, including our own, have a critical limitation: they have only one pixel. Capturing an image with these cameras requires mechanical scanning across the field of view, a slow and tedious process which places profound limits on imaging applications. For example, despite their extremely fast temporal resolution, single- pixel, single-photon cameras cannot be used to image dynamic scenes, e.g., settings where the scene or camera moves or where the illumination itself is changing. Instead, these cameras are restricted to imaging static scenes in laboratory settings where scanning and image acquisition can take minutes to hours. Moreover, current single-photon cameras have restricted bandwidth, limiting the total number of photons that can be collected within a given period of time. Motivated by the recently announced commercial availability of 2D single-photon cameras, we propose to build a novel experimental testbed to push dynamic computer vision, 2D imaging, and 3D sensing to an entirely new level of speed, sensitivity, and accuracy. Specifically, this proposal requests 2D single-photon cameras of two distinct types: (1) a single-photon camera with 23 pixels that individually detect and stream out up to 7.8 million high-precision photon arrival times per second, and (2) a high-resolution 512x512 camera that outputs 100,000 binary frames of photon detections per second. The first type of camera is a 2D generalization of single-pixel, single-photon cameras, offering a 20x improvement in photon count rate, dramatically reduced scan times, and extreme imaging capabilities: low-light, high-speed videography; dynamic, single-photon 3D imaging; and passive localization and mapping in the dark. The second type of camera is best understood as a high-frame-rate, high-resolution, zero-noise binary video camera; while it does not output photon timing information, its single-photon sensitivity, high speed, and high spatial resolution enable new forms of real-time visual processing, and may lead to revolutionary new types of Earth-based astronomical telescopes in a collaborative research direction described in the proposal.