Incremental modeling of geometry and appearance from video

研究的重点是2D图像和3D模型之间的中间结构的表示和计算方法。传统上，图像处理和特征检测首先在每个2D图像的基础上进行。然后，在单个批处理过程中，从所有2D特征计算3D几何。我们将特别探讨： (1)自由空间限制。3D曲面片的每个视图都意味着在摄影机和该曲面之间有一个自由空间体积。对这些体积进行积分将产生常规三角剖分的替代3D建模。虽然自由空间被用于机器人的2D地形图，但在3D计算机视觉中很少被探索，构建高效而准确的增量式3D体合并算法是具有挑战性的。 (2)代理几何。认为计算的几何体准确地表示真实场景的假设是脆弱的(而且往往是不正确的)。在计算机图形学的一个分支--基于图像的绘制(IBR)中，粗略的代理几何体被用来表示(索引)光线集。代理几何体可用于索引深度置换贴图[22]和外观信息。 (3)纹理/外观的统计模型。许多计算机视觉使用颜色恒定假设--每个表面点都有一种颜色。一般来说，这并不适用于物理表面，但更重要的是，算法经常失败，因为在不精确的几何图形下，颜色恒定是一个特别糟糕的假设。外观的统计模型受到的限制较少。我们率先将线性子空间模型用于纹理[Jagerand，CVPR 1997]，并继续将其用于3D建模[32，37]。这里将探索用于非线性空间和增量计算的算法。 (4)多层次模型。使用粗略的3D几何体代理来索引置换贴图，而置换贴图又索引参数化的外观模型，从而产生一个集成模型，其中不同的信息以不同的比例表示：宏观、中观和微观。

The focus of the proposed research is representations and computational methods for intermediate structures between 2D images and 3D models. Classically image processing and feature detection has been done first on a per-2D-image basis. Then in a single batch process a 3D geometry is computed from all the 2D features. In particular we will explore: (1) Free-space constraints. Each view of a 3D surface patch implies there is a free-space volume between the camera and that surface. Integrating such volumes yields an alternate 3D modeling to the conventional triangulation. While free-space is used in robotics for 2D ground maps, it is little explored in 3D computer vision, and constructing efficient and accurate incremental 3D volume merging algorithms is challenging. (2) Proxy geometries. The assumption that a computed geometry is accurately representing the real scene is fragile (and often incorrect). In Image-Based Rendering (IBR) a branch of computer graphics, a coarse proxy geometry is used to represent (index) ray-sets. A proxy geometry can be used to index depth displacement maps[22] and appearance information as well. (3) Statistical models of texture/appearance. Much computer vision uses the color-constancy assumption -- each surface point has a color. This is not true of physical surfaces in general, but more importantly algorithms often fail because color constancy is a particularly poor assumption under inexact geometries. Statistical models of appearance are less constrained. We have pioneered the use of linear subspace models for texture [Jagersand, CVPR 1997] and continued to develop this for 3D modeling [32,37]. Algorithms for non-linear spaces and incremental computation will be explored here. (4) Multi-level models. The use of a coarse 3D geometry proxy to index a displacement map, which in turn indexes a parametrized appearance model leads to an integrated model where different information is represented at different scales: Macro, Meso and Micro.