CAREER: Toward a General Framework for Words and Pictures

图片直接将世界的视觉描述传达给观众。计算机视觉致力于设计算法来提取相机眼中捕获的潜在世界状态，其总体目标是实现通用计算图像理解。迄今为止，许多视觉研究都是通过关注目标检测来解决图像理解问题，这只是图像理解问题的一个角度。这个项目着眼于一种额外的、免费的方式来收集关于视觉世界的信息——通过直接分析网络上大量的视觉描述性文本来揭示哪些信息是有用的，并从图片中提取出来。这个项目提出了一个全面的研究计划，旨在建立和利用文字和图片的互补性质。一个主要目标是研究文本和图像之间的联系，以了解描述——通过图片进行意义交流。这一目标通过3大挑战来实现：1)开发更丰富的词汇来描述描述所提供的信息。2)开发能够从视觉上捕捉这些更微妙词汇的图像表示。3)构建全面的图文联合框架。该项目对互联网上访问图像的许多具体任务具有直接意义，包括：图像搜索、浏览和组织，以及产品搜索等商业应用，以及盲人网络辅助等社会重要应用。此外，这个项目的产出，包括在视觉描述的自然词汇和结构方面的进展，对计算机视觉和自然语言社区的交叉影响都有很大的潜力。

Pictures convey a visual description of the world directly to their viewers. Computer vision strives to design algorithms to extract the underlying world state captured in the camera's eye, with an overarching goal of general computational image understanding. To date much vision research has approached image understanding by focusing on object detection, only one perspective on the image understanding problem. This project looks at an additional, complimentary way to collect information about the visual world -- by directly analyzing the enormous amount of visually descriptive text on the web to reveal what information is useful to attach to, and extract from pictures. This project presents a comprehensive research program geared toward modeling and exploiting the complimentary nature of words and pictures. One main goal is studying the connection between text and images to learn about depiction -- communication of meaning through pictures. This goal is addressed through 3 broad challenges: 1) Developing a richer vocabulary to describe the information provided by depiction. 2) Developing image representations that can visually capture this more nuanced vocabulary. 3) Constructing a comprehensive joint words and pictures framework. This project has direct significance to many concrete tasks that access images on the internet including: image search, browsing, and organization, as well as commercial applications such as product search, and societally important applications such as web assistance for the blind. Additionally, outputs of this project, including progress toward a natural vocabulary and structure for visual description, have great potential for cross-cutting impact in both the computer vision and natural language communities.