Model Selection for Surface Approximation and Scene Interpretation

运动结构（SfM）和多视图立体（MVS）方法为三维重建提供了经济实惠的手段。然而，生成的输出（通常是点云）会受到错误、缺失区域以及经常出现的严重异常值的影响。点云也是一种相当粗糙的数据形式，因为没有更高层次的几何或语义信息可以立即访问。在建筑立面的背景下，该项目寻求开发从MVS点云中获得更多功能模型的方法。我们的方法基于这样一种见解，即几何和语义正则化的结合是输入数据合理抽象的关键。我们建议在几何正则化和语义解释之间交替进行，双方相互提供信息。在得到的迭代方案中，几何侧可以利用语义信息对重构应用特定的先验。另一方面，语义解释方面，除了最初的视觉数据外，还受益于通过改进和理想化的几何抽象进行的交换。所建议的方法的输出将是输入数据的抽象。噪音、异常值和“不重要”的细节被剔除，以保留建筑立面的本质。生成模型的各个部分将具有语义意义，以标签的形式表示，如“窗口”、“阳台”和“门”。此外，建筑风格以及立面的结构（例如，窗户重复的模式）将被了解。我们相信这些方法是朝着快速和经济地生成详细的城市模型迈出的重要一步，重建不仅在视觉上准确，而且在语义上丰富。在第一个资助期的第一年，我们进行了初步测试，并制定了基本框架，其设计强烈以实验为指导。尽管到目前为止项目持续时间很短，但我们在具有挑战性的输入数据上展示了令人信服的结果，并认为进一步的开发确实是必要的。在第二个资助期，我们将集中精力改进几何正则化和语义解释以及适当的定量评估。此外，我们将改进我们的数据采集和纹理能力，以增加理想化抽象的视觉保真度。作为理想化的、视觉上准确的语义标记抽象的另一个应用场景，我们将比较重建并自动检测建筑立面上的变化，这些变化将根据我们的语义丰富的数据进行分类。

Structure from Motion (SfM) and Multi-View Stereo (MVS) methods provide affordable means for 3D reconstruction. However, the generated output, typically point clouds, are subject to errors, missing regions, and, quite often, severe outliers. The point cloud is also a rather crude form of data in the sense that no higher level geometric or semantic information is immediately accessible.In the context of building facades, this project seeks to develop methods that derive more versatile models from a MVS point cloud. We base our approach on the insight that a combination of geometric and semantic regularization is key to a plausible abstraction of the input data. We propose to alternate between geometric regularization and semantic interpretation with both sides providing information to each other. In the resulting iterative scheme, the geometric side can make use of semantic information in order to apply specific priors on the reconstruction. The semantic interpretation side, on the other hand, benefits from the exchange by means of the improved and idealized geometric abstraction in addition to the initial visual data.The output of the proposed method will be an abstraction of the input data. Noise, outliers, and "unimportant" details are culled away to retain the essence of the building facade. The individual parts of the generated model will have a semantic meaning, in the form of labels such as "window", "balcony", and "door". In addition, the architectural style as well as the structure of the facade (e.g., the pattern of window repetitions) will be known.We believe that these methods are an important step towards rapid and affordable generation of detailed city models where the reconstruction is not only visually accurate, but also semantically enriched.In the first year of the first funding period, we conducted initial testing and developed the basic framework, the design of which was strongly guided by experiments. Despite the short duration of the project so far, we demonstrate convincing results on challenging input data and argue that further development is indeed warranted. In the second funding period, we will concentrate our efforts on improved geometric regularization and semantic interpretation as well as a proper, quantitative evaluation. In addition, we will improve our data acquisition and texturing capabilities to increase the visual fidelity of the idealized abstraction. As an additional application scenario of the idealized, visually accurate semantically labeled abstraction, we will compare reconstructions and automatically detect changes on building facades that will be classified based on our semantically enriched data.