BIGDATA: Small: DA: DCM: Labeling the World

该项目旨在利用大量在线照片，文本和地图的语料库来创建世界的语义3D标记模型，例如，世界顶级文化和历史遗址的详细展示。 虽然计算机视觉的突破使我们能够从数百万张在线2D图像中创建详细的3D模型，但生成的模型只能捕获几何形状。因此，它们缺乏语义;它们不提供有关场景内容的信息。 维基百科等在线文本的巨大宝库仔细地对照片和模型中捕捉的场景进行了分类。 现代自然语言处理（NLP）技术现在可以处理这些数据，从而有机会从在线文本语料库中提取知识，并将其用于标记3D几何形状。 该项目旨在共同分析大量在线文本、地图和照片的语料库，以创建世界各地的标记3D模型。 实现这一目标将需要在自然语言处理和计算机视觉的界面上取得基础研究进展，这些进展将影响科学研究界和整个世界。该项目解决了两个关键的技术挑战：（1）自动场景标记：将语义映射到几何图形上，以及（2）解决3D拼图：将几何图形映射到世界上。 这些映射问题的许多线索都存在于文本和其他在线文档中，如平面图。 其他线索在于照片的内容。 因此，解码这种映射涉及NLP和计算机视觉之间的相互作用。 关键的研究进展围绕着联合利用计算机视觉和自然语言处理来解决问题的新方法，以解决这两个领域中的挑战性问题，具体来说，1）通过联合自然语言处理和3D视觉分析来识别物体，2）通过将几何形状与地图和网页中的空间文本相关联来将物体放置在世界上，（3）利用语义学，通过文本空间关系增强视觉线索，从而改进几何学。（1）大规模创建标记3D模型的技术，以及（2）许多顶级旅游景点的标记模型。 算法和模型都将免费提供给研究界。 这些算法和模型将为一系列对世界产生重大实际影响的令人兴奋的应用奠定基础。 由此产生的工具可以使维基百科等资源将文本直接链接到3D模型，反之亦然，并为在线学习和教育带来好处。 同样的技术可以实现2D照片的自动标记。 在实时应用的上下文中（例如，增强现实），该技术可以提供视觉覆盖和即时反馈，你正在看什么，并使增强现实风格的导游图尔斯。 其他应用包括使用带标签的几何图形进行导航（步行方向），以及将图像转换为文本以供视障人士使用。这项研究与华盛顿大学的学生教育和培训紧密结合。有关该项目的更多信息，请访问：http://grail.cs.washington.edu/projects/label3d/

The project aims to leverage the massive corpus of online photos, text, and maps to create a semantic 3D labeled model of the world, e.g., detailed representations of the world's top cultural and historical sites. While breakthroughs in computer vision enable creating detailed 3D models from millions of online 2D images, the resulting models capture only geometry. Consequently, they lack semantics; they don't provide information about the contents of the scene. The vast treasure trove of online text such as Wikipedia meticulously catalogs the scenes that are captured in photos and models. Modern Natural Language Processing (NLP) techniques can now process such data, opening up the opportunity to extract knowledge from the online text corpus and use it to label 3D geometry. This project seeks to jointly analyze the massive corpus of online text, maps, and photos to create labeled 3D models of the world's sites. Achieving this goal will require fundamental research advances at the interface of natural language processing and computer vision that impact both the scientific research community and the world at large. The project addresses two key technical challenges: (1) automatic scene labeling: mapping semantics onto geometry, and (2) solving the 3D jigsaw puzzle: mapping pieces of geometry into the world. Many clues to these mapping problems lie in the text and other online datasources such as floorplans. Other clues lie in the content of the photos. Decoding this mapping therefore involves an interplay between NLP and computer vision. The key research advances center around new ways to jointly leverage computer vision and NLP to solve problems to solve challenging problems in both fields, specifically, 1) recognizing objects through joint NLP and 3D visual analysis, 2) placing objects in the world by correlating geometry with spatial text in maps and webpages, and 3) using semantics to improve geometry by augmenting visual cues with textual spatial relations.Broader Impacts: The primary research outcomes are: (1) technology for creating labeled 3D models at a massive scale, and (2) labeled models for many top tourist sites. Both the algorithms and models will be made freely available for the research community. These algorithms and models will provide the foundation for a range of exciting applications of major practical impact on the world at large. The resulting tools could make it possible for resources such as Wikipedia to link the text directly to 3D models and vice-versa, with attendant benefits to online learning and education. The same technology could enable automated labeling of 2D photographs. In the context of real-time applications (e.g., augmented reality), the technology could provide visual overlays and instant feedback on what you are currently looking at, and enable augmented reality-style guided tours. Other applications include using labeled geometry for navigation (walking directions), and converting images to text for the visually impaired. The research is tightly integrated into education and training of students at the University of Washington. Additional information about the project can be found at: http://grail.cs.washington.edu/projects/label3d/