CAREER: Toward a General Framework for Words and Pictures

图片直接向观众传达了对世界的视觉描述。计算机视觉致力于设计算法来提取相机眼睛捕捉到的潜在世界状态，总体目标是全面的计算图像理解。到目前为止，许多视觉研究都是通过目标检测来接近图像理解的，对图像理解问题的研究只有一个角度。这个项目着眼于一种额外的、免费的方式来收集关于视觉世界的信息--通过直接分析网络上大量的视觉描述性文本来揭示哪些信息是有用的，可以附加到图片上，或者从图片中提取出来。这个项目提出了一个全面的研究计划，旨在建模和利用文字和图片的互补性质。其中一个主要目标是研究文字和图像之间的联系，以了解描述--通过图像传达意义。这一目标通过三个广泛的挑战来实现：1)开发更丰富的词汇来描述描述提供的信息。2)开发能够在视觉上捕捉到这种更加细微差别的词汇的图像表示法。3)构建全面的图文联动框架。该项目对许多访问互联网上图像的具体任务具有直接意义，包括：图像搜索、浏览和组织，以及商业应用程序(如产品搜索)和具有社会重要性的应用程序(如为盲人提供网络帮助)。此外，该项目的成果，包括在自然词汇和视觉描述结构方面取得的进展，在计算机视觉和自然语言界都具有广泛影响的巨大潜力。

Pictures convey a visual description of the world directly to their viewers. Computer vision strives to design algorithms to extract the underlying world state captured in the camera's eye, with an overarching goal of general computational image understanding. To date much vision research has approached image understanding by focusing on object detection, only one perspective on the image understanding problem. This project looks at an additional, complimentary way to collect information about the visual world -- by directly analyzing the enormous amount of visually descriptive text on the web to reveal what information is useful to attach to, and extract from pictures. This project presents a comprehensive research program geared toward modeling and exploiting the complimentary nature of words and pictures. One main goal is studying the connection between text and images to learn about depiction -- communication of meaning through pictures. This goal is addressed through 3 broad challenges: 1) Developing a richer vocabulary to describe the information provided by depiction. 2) Developing image representations that can visually capture this more nuanced vocabulary. 3) Constructing a comprehensive joint words and pictures framework. This project has direct significance to many concrete tasks that access images on the internet including: image search, browsing, and organization, as well as commercial applications such as product search, and societally important applications such as web assistance for the blind. Additionally, outputs of this project, including progress toward a natural vocabulary and structure for visual description, have great potential for cross-cutting impact in both the computer vision and natural language communities.