Discovering and Reconstructing Scenes from Photos on the Internet

摘要该SGER提案提出了一个为期一年的研究计划，旨在探索两个令人兴奋的全新方向，即(1)在互联网上提供的照片集中发现重要的场景和物体；(2)从这些在线照片集中重建密集的几何结构。互联网上发布的照片的规模、多样性和无组织性对现有的计算机视觉技术提出了重大挑战。该SGER提案描述了应对这些挑战的关键解决方案。第一个目标是设计通过识别图像集合中包含的重要场景和对象来自动推断图像集合内容的技术。例如，可以从Flickr上的近一百万张照片中推断出所有受欢迎的旅游景点、雕像、绘画和其他罗马文物。这需要计算这些场景和对象的规范视图，它们共同涵盖了这些场景中最有趣的方面。第二个目标是开发多视图立体技术（MVS），可以有效地处理这些不受控制和高度可变的图像集。由于光照、相机响应和前景杂波在不同图像之间可能存在很大差异，因此需要新的立体匹配算法。pi将制定策略来选择哪些视图进行组合，探索匹配的度量来排除照明和相机变化，并最终利用照明变化来结合光度和几何立体。最终，这项研究可以创建一个几何爬虫，它可以在互联网上搜索物体来重建，这样的方法可以被使用，给定足够的计算时间和计算能力，为世界上所有拍摄良好的地点、城市、景观和物体自动创建3D模型。本研究的成果包括可以自动发现和描述场景以及从互联网集合中重建几何模型的工具。这一结果将使一系列重要的应用成为可能，包括3D可视化、定位、通信和识别，这些应用远远超出了传统的计算机视觉问题，并可能对整个人群产生广泛的影响。此外，拟议的工作将为一系列受众创造大量的新资源。首先，研究的输出将是世界上许多地点的大量注册图像数据集和这些相同地点的密集3D重建。这些数据将广泛分发，以帮助推进计算机视觉社区的研究。这些数据还将用于许多其他目的，例如基于图像的渲染、文化遗产、本地化工作和科学可视化的计算机图形学研究。

AbstractThis SGER proposal sets forth a one year research plan to explore two exciting and fundamentally new directions, namely (1) discovering significant scenes and objects in photo collections available on the internet and (2) reconstructing dense geometry from these online photo collections.The scale, diversity, and unorganized nature of photos posted on the Internet present major challenges to existing computer vision techniques. This SGER proposal describes key solutions to address these challenges. The first goal is to devise techniques to automatically infer the content of image collections, by identifying the significant scenes and objects that are contained therein. For example, it may be possible to deduce all of the popular tourist sites, statues, paintings, and other artifacts of Rome from the nearly one million photos on Flickr. This requires having to compute canonical views of these scenes and objects, that together cover the most interesting aspects of these scenes.The second goal is to develop multi-view stereo techniques (MVS) that can operate effectively on such uncontrolled and highly variable image sets. Because lighting, camera response, and foreground clutter can differ substantially from image to image, new stereo matching algorithms are required. The PIs will develop strategies for selecting which views to combine, explore matching metrics to factor out lighting and camera variations, and ultimately leverage lighting variations to combine photometric and geometric stereo.Ultimately, this research can lead to the creation of a geometry crawler that scours the Internet for objects to reconstruct such an approach could be used, given sufficient compute time and compute power, to automatically create 3D models for all of the world's well- photographed sites, cities, landscapes, and objects.The outcome of this research consists of tools that can automatically discover and describe scenes and reconstruct geometric models from Internet collections. This outcome will enable a host of important applications, ranging across 3D visualization, localization, communication, and recognition, that go well beyond traditional computer vision problems and can have broad impacts for the population at large.In addition, the proposed work will create a large set of new resources for a range of audiences. First, the output of the research will be massive datasets of registered imagery for many world sites and dense 3D reconstructions of those same sites. This data will be distributed broadly to help advance research in the computer vision community. The data will also be made available for many other purposes, such as computer graphics research into image-based rendering, cultural heritage, localization efforts, and scientific visualization.