ITR: High Performance Imaging Using an Array of Low-Cost Cameras

该项目将探索128个摄像头阵列的功能，这些摄像头可以在许多不同的配置中以计算方式组合在一起，用于广泛的科学、商业和通信应用。在过去的几年里，基于图像的绘制(IBR)技术能够生成真实场景的照片级真实感图像，这引起了人们对构建能够从多个视角捕捉环境的传感器系统的极大兴趣。与此同时，我们见证了CMOS型图像传感器的出现，由于其数字接口，这种传感器价格低廉，使用方便。此外，因为它们是在cmos工艺中制造的，所以处理能力可以施加到传感器本身。最后，半导体技术的进步使越来越多的计算能力可用于降低成本、功耗和芯片面积。这些趋势提出了这样一个问题：我们能用这么多便宜的cmos图像传感器、同样便宜的光学元件和强大的处理能力来做什么？我们可以使用更多质量较差的摄像头来实现更强大的IBR算法吗？我们能使用廉价的成像器和处理器集群来创建性能优于真实相机的虚拟相机吗？128个相机阵列中的每个相机都包含一个CMOS图像传感器、MPEG编码器和可编程处理器，以研究这些问题。该设备旨在通过三台PC将128个同步视频数据集记录到一个磁盘阵列上。该项目将探索该阵列在科学成像、计算机视觉和图形学方面的应用。多摄像头系统可以在许多方面发挥作用。如果摄像机紧密组合在一起，则系统可有效地充当单中心投影合成摄像机，可配置为在一个或多个成像维度上提供高性能，例如分辨率、信噪比、动态范围、景深、帧速率或光谱灵敏度。例如，一种配置可以生成10,240 x 3,830像素的高分辨率图像，而另一种配置可以每秒生成7,680帧。这种能力对于视频系统来说是前所未有的，它们将有许多科学、工程和军事用途。如果相机被放置得更远，那么系统的功能就像一个多中心投影相机，它捕获的数据称为光场。特别令人感兴趣的是用于从阵列捕获的密集图像估计3D场景几何形状的新方法。该信息可用于改进光场的压缩，并在大范围间隔的摄影机之间平滑地进行内插，从而允许在场景中平滑地进行虚拟导航。潜在的应用包括评估制造、医疗和法医咨询、在线购物和虚拟博物馆展示的设计模型。

This project will explore the capabilities of an array of 128 cameras, which can be combined computationally in many different configurations for a wide range of scientific, commercial, and communication applications. Over the past few years, the ability of image-based rendering (IBR) techniques to create photorealistic images of real scenes has generated great interest in building sensor systems that can capture environments from multiple viewpoints. At the same time, we have witnessed the advent of CMOS image sensors, which are inexpensive and easy to use because of their digital interface. Furthermore, because they are manufactured in a CMOS process, processing power can be placed on the sensors themselves. Finally, advances in semiconductor technology are making increasing computing power available for decreasing cost, power, and chip area. These trends raise the questions: What can we do with many inexpensive CMOS image sensors, equally inexpensive optics, and a lot of processing power? Can we use more cameras of lesser quality to enable more robust IBR algorithms? Can we use clusters of inexpensive imagers and processors to create virtual cameras that outperform real ones?Each camera in the 128-camera array contains a CMOS image sensor, MPEG encoder, and programmable processor, in order to investigate these questions. The device is designed to record 128 synchronized video datasets through three PCs to a disk array. This project will explore applications of the array to scientific imaging and computer vision and graphics. Multi-camera systems can function in many ways. If the cameras are packed close together, then the system effectively functions as a single-center-of-projection synthetic camera, which can be configured to provide high performance along one or more imaging dimensions, such as resolution, signal-to-noise ratio, dynamic range, depth of field, frame rate, or spectral sensitivity. For example, one configuration could produce high-resolution images 10,240 x 3,830 pixels, and another could generate 7,680 frames per second. Such capabilities are unprecedented for a video system, and they will have many scientific, engineering, and military uses. If the cameras are placed farther apart, then the system functions as a multiple-center-of-projection camera, and the data it captures is called a light field. Of particular interest are novel methods for estimating 3D scene geometry from the dense imagery captured by the array. This information can be used to improve compression of the light field and to interpolate smoothly between widely spaced cameras, allowing smooth virtual navigation through the scene. Potential applications include evaluation of design models for manufacturing, medical and forensic consultation, online shopping, and virtual museum displays.