BIGDATA: Small: DA: DCM: Labeling the World

点击翻译按钮获取中文摘要

The project aims to leverage the massive corpus of online photos, text, and maps to create a semantic 3D labeled model of the world, e.g., detailed representations of the world's top cultural and historical sites. While breakthroughs in computer vision enable creating detailed 3D models from millions of online 2D images, the resulting models capture only geometry. Consequently, they lack semantics; they don't provide information about the contents of the scene. The vast treasure trove of online text such as Wikipedia meticulously catalogs the scenes that are captured in photos and models. Modern Natural Language Processing (NLP) techniques can now process such data, opening up the opportunity to extract knowledge from the online text corpus and use it to label 3D geometry. This project seeks to jointly analyze the massive corpus of online text, maps, and photos to create labeled 3D models of the world's sites. Achieving this goal will require fundamental research advances at the interface of natural language processing and computer vision that impact both the scientific research community and the world at large. The project addresses two key technical challenges: (1) automatic scene labeling: mapping semantics onto geometry, and (2) solving the 3D jigsaw puzzle: mapping pieces of geometry into the world. Many clues to these mapping problems lie in the text and other online datasources such as floorplans. Other clues lie in the content of the photos. Decoding this mapping therefore involves an interplay between NLP and computer vision. The key research advances center around new ways to jointly leverage computer vision and NLP to solve problems to solve challenging problems in both fields, specifically, 1) recognizing objects through joint NLP and 3D visual analysis, 2) placing objects in the world by correlating geometry with spatial text in maps and webpages, and 3) using semantics to improve geometry by augmenting visual cues with textual spatial relations.Broader Impacts: The primary research outcomes are: (1) technology for creating labeled 3D models at a massive scale, and (2) labeled models for many top tourist sites. Both the algorithms and models will be made freely available for the research community. These algorithms and models will provide the foundation for a range of exciting applications of major practical impact on the world at large. The resulting tools could make it possible for resources such as Wikipedia to link the text directly to 3D models and vice-versa, with attendant benefits to online learning and education. The same technology could enable automated labeling of 2D photographs. In the context of real-time applications (e.g., augmented reality), the technology could provide visual overlays and instant feedback on what you are currently looking at, and enable augmented reality-style guided tours. Other applications include using labeled geometry for navigation (walking directions), and converting images to text for the visually impaired. The research is tightly integrated into education and training of students at the University of Washington. Additional information about the project can be found at: http://grail.cs.washington.edu/projects/label3d/