Single-Photon Cameras for Extreme Computer Vision and Computational Astronomy

照相机通过收集光子来捕捉图像：光子是单独、随机和异步到达传感器的离散光包。虽然这个物理过程被绝大多数计算机视觉算法抽象出来，但新兴的单光子相机提供了精确检测和记录单个光子到达时间的方法。在过去的五年里，单光子相机的日益普及带来了许多前所未有的成像功能，从可以在近乎完全黑暗和角落周围看到的视觉系统，到用于汽车和智能手机的高精度3D相机，再到用于皮肤以下的新生物医学成像技术。然而，包括我们自己的实验室在内的世界各地的实验室目前使用的单光子相机有一个关键的限制：它们只有一个像素。使用这些相机捕获图像需要在整个视场中进行机械扫描，这是一个缓慢而繁琐的过程，对成像应用造成了深刻的限制。例如，尽管单像素、单光子相机具有极快的时间分辨率，但它们不能用于对动态场景进行成像，例如，设置场景或摄影机移动的位置或照明本身正在更改的位置。相反，这些相机仅限于在实验室环境中对静态场景进行成像，扫描和图像采集可能需要数分钟至数小时。此外，目前的单光子相机具有有限的带宽，限制了在给定时间段内可以收集的光子总数。受最近宣布的2D单光子相机的商业可用性的启发，我们建议建立一个新的实验测试平台，将动态计算机视觉，2D成像和3D传感推向一个全新的速度，灵敏度和准确性水平。具体而言，该提案要求两种不同类型的2D单光子相机：（1）具有23个像素的单光子相机，每秒可单独检测并输出高达780万个高精度光子到达时间，以及（2）高分辨率512 x512相机，每秒输出100，000个二进制光子检测帧。第一种类型的相机是单像素、单光子相机的2D概括，光子计数率提高20倍，扫描时间大大减少，并具有极端的成像能力：低光、高速摄像;动态、单光子3D成像;以及黑暗中的被动定位和映射。第二种相机最好理解为高帧率、高分辨率、零噪声的二进制视频相机;虽然它不输出光子定时信息，但其单光子灵敏度、高速度和高空间分辨率使新形式的实时视觉处理成为可能，并可能在提案中描述的合作研究方向上导致革命性的新型地球天文望远镜。

Cameras capture images by collecting photons: discrete packets of light that arrive to the sensor individually, stochastically, and asynchronously. While this physical process is abstracted away by the vast majority of computer vision algorithms, an emerging class of single-photon cameras provides means to precisely detect and catalog the arrival times of individual photons. Within the past five years, the growing availability of single-photon cameras has led to numerous never-seen-before imaging capabilities, from vision systems that can see in near-total darkness and around corners, to high-precision 3D cameras for automobiles and smartphones, to new biomedical imaging techniques for seeing below the skin. However, the single-photon cameras currently used by various laboratories around the world, including our own, have a critical limitation: they have only one pixel. Capturing an image with these cameras requires mechanical scanning across the field of view, a slow and tedious process which places profound limits on imaging applications. For example, despite their extremely fast temporal resolution, single- pixel, single-photon cameras cannot be used to image dynamic scenes, e.g., settings where the scene or camera moves or where the illumination itself is changing. Instead, these cameras are restricted to imaging static scenes in laboratory settings where scanning and image acquisition can take minutes to hours. Moreover, current single-photon cameras have restricted bandwidth, limiting the total number of photons that can be collected within a given period of time. Motivated by the recently announced commercial availability of 2D single-photon cameras, we propose to build a novel experimental testbed to push dynamic computer vision, 2D imaging, and 3D sensing to an entirely new level of speed, sensitivity, and accuracy. Specifically, this proposal requests 2D single-photon cameras of two distinct types: (1) a single-photon camera with 23 pixels that individually detect and stream out up to 7.8 million high-precision photon arrival times per second, and (2) a high-resolution 512x512 camera that outputs 100,000 binary frames of photon detections per second. The first type of camera is a 2D generalization of single-pixel, single-photon cameras, offering a 20x improvement in photon count rate, dramatically reduced scan times, and extreme imaging capabilities: low-light, high-speed videography; dynamic, single-photon 3D imaging; and passive localization and mapping in the dark. The second type of camera is best understood as a high-frame-rate, high-resolution, zero-noise binary video camera; while it does not output photon timing information, its single-photon sensitivity, high speed, and high spatial resolution enable new forms of real-time visual processing, and may lead to revolutionary new types of Earth-based astronomical telescopes in a collaborative research direction described in the proposal.