Camera Array for Hyperspectral Video Imaging Using Cross-Spectral Multi-View Fusion

高光谱成像记录场景中每个像素的采样光谱。高光谱成像和视频成像在农业、化学和医学等领域有着广泛的应用。高光谱成像的挑战在于，三维立方体必须使用二维传感器记录，因为每个小波长区域都记录了灰度图像。因此，这个三维图像立方体必须展开。这可以通过使用时间维度来实现，但是，使用这种方法会丢失捕获视频的能力。因此，在一次拍摄中记录高光谱数据立方体的快照光谱成像仪是非常需要的。使用现成硬件的一种方法是使用交叉光谱相机阵列。在这里，每个波长区域由一台摄像机记录，该摄像机在镜头前配备了合适的带通滤波器。随后，注册和重建过程是必要的扭曲所有相机视图到一个单一的视图。本项目的目标是建立一个37通道的高光谱相机阵列，其中配准和重建管道将完全由端到端的神经网络取代。使用神经网络，一个挑战是为训练创建足够的数据，这将通过构建高光谱渲染器和生成合成的高光谱序列来解决。一旦硬件和端到端神经网络建立和训练完毕，一个高光谱视频数据库将被创建和发布，这对科学界来说是一个新鲜事物。另一个挑战来自这样一个事实，即当穿越光谱的不同部分时，需要具有不同物理特性的不同相机。因此，通过调整端到端神经网络来支持不同物理性质的相机，解决了多设备交叉光谱问题。并对不同镜头组合下的相机排列进行了优化。

Hyperspectral imaging records a sampled light spectrum for every pixel of a scene. There are a number of applications using hyperspectral imaging and video imaging, e.g. in agriculture, chemistry, and medicine. The challenge with hyperspectral imaging is that a three dimensional cube has to be recorded using two dimension sensors, since a grayscale image is recorded for every small wavelength area. Thus, this three dimensional image cube has to be unfolded. This can be done by using the temporal dimension, however, using this approach the ability of capturing videos is lost. Therefore, snapshot spectral imagers, which record the hyperspectral data cube in one shot, are highly desirable. One approach using off-the-shelf hardware for this is by usinga cross-spectral camera array. Here, each wavelength area is recorded by one camera, which is equipped with a suitable bandpass filter in front of the lens. Subsequently, a registration and reconstruction process is necessary to warp all camera views to one single view.The goal of this project is to setup a hyperspectral camera array using 37 channels, where the registration and reconstruction pipeline shall be entirely replaced by an end-to-end neural network. Using neural networks, a challenge is to create enough data for training, which will be tackled by building a hyperspectral renderer and generating synthetic hyperspectral sequences. As soon as the hardware and the end-to-end neural networks are set up and trained, a hyperspectral video database will be created and published, which is a novelty to the scientific community. Another challenge results from the fact that different cameras with varying physical properties are necessary when traversing different parts of the light spectrum. Consequently, the multi-device cross-spectral problem is tackled by adapting the end-to-end neural networks to support camera with different physical properties. Moreover, the camera arrangement with different camera-lens combinations is optimized.