ITR: High Performance Imaging Using an Array of Low-Cost Cameras

该项目将探索128个摄像机阵列的功能，这些摄像机可以在许多不同的配置中进行计算组合，用于广泛的科学，商业和通信应用。 在过去的几年中，基于图像的渲染（IBR）技术创建真实的场景的逼真图像的能力已经在构建能够从多个视点捕获环境的传感器系统方面产生了极大的兴趣。 与此同时，我们见证了CMOS图像传感器的出现，由于其数字接口，价格低廉且易于使用。此外，由于它们是在CMOS工艺中制造的，因此处理能力可以放在传感器本身上。 最后，半导体技术的进步使得越来越多的计算能力可用于降低成本、功率和芯片面积。 这些趋势提出了一个问题：我们可以用许多廉价的CMOS图像传感器、同样廉价的光学器件和强大的处理能力来做什么？ 我们是否可以使用更多质量较低的摄像头来实现更强大的IBR算法？ 我们能否使用廉价的成像器和处理器集群来创建性能优于真实的虚拟相机？为了研究这些问题，128个摄像机阵列中的每个摄像机都包含CMOS图像传感器、MPEG编码器和可编程处理器。 该设备设计用于通过三台PC将128个同步视频数据集记录到磁盘阵列。 该项目将探索阵列在科学成像、计算机视觉和图形学方面的应用。多相机系统可以以多种方式工作。 如果照相机紧密地包装在一起，则系统有效地用作单投影中心合成照相机，其可以被配置为沿着一个或多个成像维度（诸如分辨率、信噪比、动态范围、景深、帧速率或光谱灵敏度）提供高性能沿着。例如，一种配置可以生成10，240 x 3，830像素的高分辨率图像，另一种配置可以每秒生成7，680帧。 这种能力对于视频系统来说是前所未有的，它们将有许多科学，工程和军事用途。如果两个摄像机之间的距离更远，那么系统就像一个多中心投影摄像机，它捕获的数据被称为光场。 特别感兴趣的是用于从由阵列捕获的密集图像估计3D场景几何形状的新方法。 该信息可用于改善光场的压缩并在间距很大的相机之间平滑插值，从而允许在场景中平滑虚拟导航。 潜在的应用包括评估制造，医疗和法医咨询，网上购物和虚拟博物馆展示的设计模型。

This project will explore the capabilities of an array of 128 cameras, which can be combined computationally in many different configurations for a wide range of scientific, commercial, and communication applications. Over the past few years, the ability of image-based rendering (IBR) techniques to create photorealistic images of real scenes has generated great interest in building sensor systems that can capture environments from multiple viewpoints. At the same time, we have witnessed the advent of CMOS image sensors, which are inexpensive and easy to use because of their digital interface. Furthermore, because they are manufactured in a CMOS process, processing power can be placed on the sensors themselves. Finally, advances in semiconductor technology are making increasing computing power available for decreasing cost, power, and chip area. These trends raise the questions: What can we do with many inexpensive CMOS image sensors, equally inexpensive optics, and a lot of processing power? Can we use more cameras of lesser quality to enable more robust IBR algorithms? Can we use clusters of inexpensive imagers and processors to create virtual cameras that outperform real ones?Each camera in the 128-camera array contains a CMOS image sensor, MPEG encoder, and programmable processor, in order to investigate these questions. The device is designed to record 128 synchronized video datasets through three PCs to a disk array. This project will explore applications of the array to scientific imaging and computer vision and graphics. Multi-camera systems can function in many ways. If the cameras are packed close together, then the system effectively functions as a single-center-of-projection synthetic camera, which can be configured to provide high performance along one or more imaging dimensions, such as resolution, signal-to-noise ratio, dynamic range, depth of field, frame rate, or spectral sensitivity. For example, one configuration could produce high-resolution images 10,240 x 3,830 pixels, and another could generate 7,680 frames per second. Such capabilities are unprecedented for a video system, and they will have many scientific, engineering, and military uses. If the cameras are placed farther apart, then the system functions as a multiple-center-of-projection camera, and the data it captures is called a light field. Of particular interest are novel methods for estimating 3D scene geometry from the dense imagery captured by the array. This information can be used to improve compression of the light field and to interpolate smoothly between widely spaced cameras, allowing smooth virtual navigation through the scene. Potential applications include evaluation of design models for manufacturing, medical and forensic consultation, online shopping, and virtual museum displays.