Model Selection for Surface Approximation and Scene Interpretation

运动结构(SfM)和多视点立体(MVS)方法为三维重建提供了负担得起的手段。然而，生成的输出(通常是点云)容易出现错误、缺失区域，而且通常还会出现严重的异常值。点云也是一种相当原始的数据形式，因为无法立即获得更高级别的几何或语义信息。在建筑立面的背景下，该项目寻求开发从MVS点云获得更多通用模型的方法。我们的方法基于这样一种见解，即几何和语义规则化的组合是输入数据合理抽象的关键。我们建议在几何正则化和语义解释之间交替进行，双方相互提供信息。在所得到的迭代方案中，几何方可以利用语义信息来对重建应用特定的先验。另一方面，语义解释方面通过改进和理想化的几何抽象在原始视觉数据的基础上进行交换，所提出的方法的输出将是输入数据的抽象。噪声、离群值和“不重要”的细节被剔除，以保留建筑立面的精髓。生成的模型的各个部分将具有语义含义，其形式为“窗”、“阳台”和“门”。此外，建筑风格以及立面的结构(例如，窗户重复的模式)将是已知的。我们相信，这些方法是朝着快速和负担得起的详细城市模型迈出的重要一步，重建不仅在视觉上准确，而且在语义上也丰富。在第一个资助期的第一年，我们进行了初步测试并开发了基本框架，其设计受到实验的强烈指导。尽管到目前为止该项目的持续时间很短，但我们在挑战输入数据方面展示了令人信服的结果，并辩称确实有必要进一步发展。在第二个资助期，我们将集中精力改进几何正则化和语义解释，以及适当的、定量的评估。此外，我们将改进我们的数据采集和纹理处理能力，以提高理想化抽象的视觉保真度。作为理想化的、视觉上准确的语义标签抽象的另一个应用场景，我们将比较重构并自动检测将基于丰富的语义数据进行分类的建筑外观上的更改。

Structure from Motion (SfM) and Multi-View Stereo (MVS) methods provide affordable means for 3D reconstruction. However, the generated output, typically point clouds, are subject to errors, missing regions, and, quite often, severe outliers. The point cloud is also a rather crude form of data in the sense that no higher level geometric or semantic information is immediately accessible.In the context of building facades, this project seeks to develop methods that derive more versatile models from a MVS point cloud. We base our approach on the insight that a combination of geometric and semantic regularization is key to a plausible abstraction of the input data. We propose to alternate between geometric regularization and semantic interpretation with both sides providing information to each other. In the resulting iterative scheme, the geometric side can make use of semantic information in order to apply specific priors on the reconstruction. The semantic interpretation side, on the other hand, benefits from the exchange by means of the improved and idealized geometric abstraction in addition to the initial visual data.The output of the proposed method will be an abstraction of the input data. Noise, outliers, and "unimportant" details are culled away to retain the essence of the building facade. The individual parts of the generated model will have a semantic meaning, in the form of labels such as "window", "balcony", and "door". In addition, the architectural style as well as the structure of the facade (e.g., the pattern of window repetitions) will be known.We believe that these methods are an important step towards rapid and affordable generation of detailed city models where the reconstruction is not only visually accurate, but also semantically enriched.In the first year of the first funding period, we conducted initial testing and developed the basic framework, the design of which was strongly guided by experiments. Despite the short duration of the project so far, we demonstrate convincing results on challenging input data and argue that further development is indeed warranted. In the second funding period, we will concentrate our efforts on improved geometric regularization and semantic interpretation as well as a proper, quantitative evaluation. In addition, we will improve our data acquisition and texturing capabilities to increase the visual fidelity of the idealized abstraction. As an additional application scenario of the idealized, visually accurate semantically labeled abstraction, we will compare reconstructions and automatically detect changes on building facades that will be classified based on our semantically enriched data.