Single-Photon Cameras for Extreme Computer Vision and Computational Astronomy

相机通过收集光子来捕捉图像：单独、随机和异步地到达传感器的离散光分组。虽然这一物理过程被绝大多数计算机视觉算法抽象掉了，但一种新兴的单光子相机提供了精确检测和分类单个光子到达时间的方法。在过去的五年里，单光子相机的日益普及带来了许多前所未有的成像能力，从可以在几乎完全黑暗的环境中和角落看到的视觉系统，到汽车和智能手机的高精度3D相机，再到新的生物医学成像技术，可以看到皮肤下的情况。然而，目前世界各地各种实验室使用的单光子相机，包括我们自己的实验室，都有一个严重的限制：它们只有一个像素。使用这些相机拍摄图像需要在整个视场中进行机械扫描，这是一个缓慢而乏味的过程，对成像应用造成了严重限制。例如，尽管它们的时间分辨率极快，但单像素、单光子相机不能用于对动态场景成像，例如场景或相机移动的设置或照明本身正在改变的设置。取而代之的是，这些相机仅限于在实验室环境中拍摄静态场景，在实验室环境中，扫描和图像采集可能需要几分钟到几个小时。此外，目前的单光子相机带宽有限，限制了在给定时间段内可以收集的光子总数。受到最近宣布的2D单光子相机商业化的激励，我们提议建立一个新型的实验试验台，将动态计算机视觉、2D成像和3D传感提升到一个全新的速度、灵敏度和精度水平。具体地说，这项提议要求两种不同类型的2D单光子摄像机：(1)23像素的单光子摄像机，分别探测和输出每秒780万个高精度光子到达时间；(2)每秒输出100,000个二进制光子探测帧的高分辨率512x512摄像机。第一种类型的相机是单像素、单光子相机的2D推广，光子计数率提高了20倍，扫描时间显著减少，成像能力极强：微光、高速摄像；动态单光子3D成像；以及黑暗中的被动定位和测绘。第二种类型的相机最好被理解为高帧频、高分辨率、零噪声的二元摄像机；虽然它不输出光子计时信息，但其单光子灵敏度、高速和高空间分辨率使新形式的实时视觉处理成为可能，并可能导致革命性的新型地面天文望远镜朝着提案中描述的合作研究方向发展。

Cameras capture images by collecting photons: discrete packets of light that arrive to the sensor individually, stochastically, and asynchronously. While this physical process is abstracted away by the vast majority of computer vision algorithms, an emerging class of single-photon cameras provides means to precisely detect and catalog the arrival times of individual photons. Within the past five years, the growing availability of single-photon cameras has led to numerous never-seen-before imaging capabilities, from vision systems that can see in near-total darkness and around corners, to high-precision 3D cameras for automobiles and smartphones, to new biomedical imaging techniques for seeing below the skin. However, the single-photon cameras currently used by various laboratories around the world, including our own, have a critical limitation: they have only one pixel. Capturing an image with these cameras requires mechanical scanning across the field of view, a slow and tedious process which places profound limits on imaging applications. For example, despite their extremely fast temporal resolution, single- pixel, single-photon cameras cannot be used to image dynamic scenes, e.g., settings where the scene or camera moves or where the illumination itself is changing. Instead, these cameras are restricted to imaging static scenes in laboratory settings where scanning and image acquisition can take minutes to hours. Moreover, current single-photon cameras have restricted bandwidth, limiting the total number of photons that can be collected within a given period of time. Motivated by the recently announced commercial availability of 2D single-photon cameras, we propose to build a novel experimental testbed to push dynamic computer vision, 2D imaging, and 3D sensing to an entirely new level of speed, sensitivity, and accuracy. Specifically, this proposal requests 2D single-photon cameras of two distinct types: (1) a single-photon camera with 23 pixels that individually detect and stream out up to 7.8 million high-precision photon arrival times per second, and (2) a high-resolution 512x512 camera that outputs 100,000 binary frames of photon detections per second. The first type of camera is a 2D generalization of single-pixel, single-photon cameras, offering a 20x improvement in photon count rate, dramatically reduced scan times, and extreme imaging capabilities: low-light, high-speed videography; dynamic, single-photon 3D imaging; and passive localization and mapping in the dark. The second type of camera is best understood as a high-frame-rate, high-resolution, zero-noise binary video camera; while it does not output photon timing information, its single-photon sensitivity, high speed, and high spatial resolution enable new forms of real-time visual processing, and may lead to revolutionary new types of Earth-based astronomical telescopes in a collaborative research direction described in the proposal.