Multimodal sparsity models for segmentation of visual objects

三维场景中物体的自动分割是计算机视觉中的一个基本问题。它是目标识别、语义场景分析、三维目标重建等许多高级算法的核心内容。到目前为止，非监督分割面临着许多挑战，如真实场景中复杂的纹理和光照变化。所提出的项目基于这样的合理假设，即对象由通过不同模式捕获的信号的特征属性来确定。我们打算证明，严格的多模式分割方法的数学推导将导致非监督分割的显著改进，最终允许对场景中的物理对象进行稳健而精确的分割。更具体地说，我们将专注于以下挑战：-受同时通过多种不同信号模式感知环境的生物系统的启发，我们打算设计算法来融合来自不同模式的感觉信息，以便大幅提高非监督分割方法的性能。-与在更高处理级别上组合信息的现有融合方案不同，我们将专注于在信号级别上跨多个模式的融合。-我们将把现有的稀疏表示和推理技术从单峰设置推广到多模式设置，并证明对多个通道的稀疏性和相互依赖性的准确建模允许更好地区分感兴趣的对象。我们的方法将足够通用，可以处理不同类型的可视数据及其跨模式依赖关系，而不需要手工制作特定于信号的特征。我们一般研究稀疏信号表示，并特别关注所谓的联合稀疏分析模型。-我们将推导出变分分割算法，该算法利用多峰稀疏性来分割无监督对象，从而将多峰稀疏性与强大的凸正则方法相结合。-我们将开发一个演示程序，它将从从场景的多个视图获取的多峰数据生成语义3D分割。我们希望证明所提出的方法对真实世界环境中的噪声和具有挑战性的光照条件具有较强的鲁棒性。

Automatic segmentation of objects in a 3D scene is a fundamental problem in Computer Vision. It is a core aspect for many high-level algorithms including object recognition, semantic scene analysis and 3D object reconstruction. To date unsupervised segmentation faces many challenges such as complex texture and lighting variations in real-world scenes. The proposed project is based on the reasonable assumption that an object is determined by characteristic properties of signals captured through different modalities. We intend to demonstrate that a rigorous mathematical derivation of multimodal segmentation approaches will lead to drastic improvements in unsupervised segmentation ultimately allowing for a robust and precise segmentation of physical objects in the scene. More specifically we will focus on the following challenges: - Inspired by biological systems which perceive their environment through many different signal modalities at once, we intend to devise algorithms to fuse sensory information from different modalities in order to drastically enhance the performance of unsupervised segmentation methods. - Unlike existing fusion schemes that combine information on higher processing levels, we will focus on fusion across several modalities on the signal level.- We will generalize existing techniques for sparse representation and inference from the unimodal setting to the multimodal setting and demonstrate that an accurate modeling of sparsity and interdependency of the multiple channels allows for much better discrimination of objects of interest. Our methods will be general enough to work on different types of visual data and their cross-modal dependencies, without the need of hand-crafted signal-specific features. We investigate sparse signal representations in general and focus on the so-called co-sparse analysis model in particular. - We will derive variational segmentation algorithms which exploit multimodal sparsity for unsupervised object segmentation, thereby combining multimodal sparsity with powerful convex regularization methods.- We will develop a demonstrator which generates semantic 3D segmentations from multimodal data taken from multiple views of a scene. We hope to demonstrate that the proposed method is robust to noise and challenging lighting conditions of real-world environments.