ITR: High Performance Imaging Using an Array of Low-Cost Cameras

该项目将探索128个摄像头阵列的能力，这些摄像头可以在许多不同的配置中进行计算组合，用于广泛的科学、商业和通信应用。在过去的几年中，基于图像的渲染（IBR）技术创建真实场景的逼真图像的能力引起了人们对构建能够从多个视点捕获环境的传感器系统的极大兴趣。与此同时，我们见证了CMOS图像传感器的出现，由于其数字接口，它价格低廉且易于使用。此外，由于它们是用CMOS工艺制造的，处理能力可以放在传感器本身上。最后，半导体技术的进步使得越来越多的计算能力可以用于降低成本、功耗和芯片面积。这些趋势提出了一个问题：我们可以用许多便宜的CMOS图像传感器，同样便宜的光学器件和大量的处理能力做什么？我们是否可以使用更多质量较低的相机来实现更强大的IBR算法？我们是否可以使用廉价的成像仪和处理器集群来创建性能优于真实相机的虚拟相机？128个摄像头阵列中的每个摄像头都包含一个CMOS图像传感器、MPEG编码器和可编程处理器，以便研究这些问题。该设备旨在通过三台pc将128个同步视频数据集记录到磁盘阵列。该项目将探索该阵列在科学成像、计算机视觉和图形学方面的应用。多摄像头系统可以以多种方式发挥作用。如果将摄像头紧密地放在一起，那么系统就可以有效地作为一个单中心投影合成摄像头，它可以配置为在一个或多个成像维度上提供高性能，例如分辨率、信噪比、动态范围、景深、帧率或光谱灵敏度。例如，一种配置可以生成10,240 x 3,830像素的高分辨率图像，而另一种配置可以每秒生成7,680帧。这样的能力对于视频系统来说是前所未有的，它们将有许多科学、工程和军事用途。如果摄像头放置得更远，那么系统就像一个多中心投影摄像头，它捕获的数据被称为光场。特别感兴趣的是从阵列捕获的密集图像中估计3D场景几何形状的新方法。这些信息可以用来改善光场的压缩，并在宽间隔的相机之间平滑地插值，从而允许在场景中进行平滑的虚拟导航。潜在的应用包括制造业、医疗和法医咨询、在线购物和虚拟博物馆展示的设计模型评估。

This project will explore the capabilities of an array of 128 cameras, which can be combined computationally in many different configurations for a wide range of scientific, commercial, and communication applications. Over the past few years, the ability of image-based rendering (IBR) techniques to create photorealistic images of real scenes has generated great interest in building sensor systems that can capture environments from multiple viewpoints. At the same time, we have witnessed the advent of CMOS image sensors, which are inexpensive and easy to use because of their digital interface. Furthermore, because they are manufactured in a CMOS process, processing power can be placed on the sensors themselves. Finally, advances in semiconductor technology are making increasing computing power available for decreasing cost, power, and chip area. These trends raise the questions: What can we do with many inexpensive CMOS image sensors, equally inexpensive optics, and a lot of processing power? Can we use more cameras of lesser quality to enable more robust IBR algorithms? Can we use clusters of inexpensive imagers and processors to create virtual cameras that outperform real ones?Each camera in the 128-camera array contains a CMOS image sensor, MPEG encoder, and programmable processor, in order to investigate these questions. The device is designed to record 128 synchronized video datasets through three PCs to a disk array. This project will explore applications of the array to scientific imaging and computer vision and graphics. Multi-camera systems can function in many ways. If the cameras are packed close together, then the system effectively functions as a single-center-of-projection synthetic camera, which can be configured to provide high performance along one or more imaging dimensions, such as resolution, signal-to-noise ratio, dynamic range, depth of field, frame rate, or spectral sensitivity. For example, one configuration could produce high-resolution images 10,240 x 3,830 pixels, and another could generate 7,680 frames per second. Such capabilities are unprecedented for a video system, and they will have many scientific, engineering, and military uses. If the cameras are placed farther apart, then the system functions as a multiple-center-of-projection camera, and the data it captures is called a light field. Of particular interest are novel methods for estimating 3D scene geometry from the dense imagery captured by the array. This information can be used to improve compression of the light field and to interpolate smoothly between widely spaced cameras, allowing smooth virtual navigation through the scene. Potential applications include evaluation of design models for manufacturing, medical and forensic consultation, online shopping, and virtual museum displays.