CAREER: Toward a General Framework for Words and Pictures

职业：建立文字和图片的通用框架

• 批准号: 1054133
• 负责人: Tamara Berg
• 金额: $ 45.9万
• 依托单位: SUNY at Stony Brook
• 依托单位国家: 美国
• 项目类别: Continuing Grant
• 财政年份: 2011
• 资助国家: 美国
• 起止时间: 2011-06-01 至 2014-06-30
• 项目状态: 已结题
• 来源: https://www.nsf.gov/awardsearch/showAward?AWD_ID=1054133&HistoricalAwards=false
• 关键词: CAREER; Toward; General; Framework; Words

Pictures convey a visual description of the world directly to their viewers. Computer vision strives to design algorithms to extract the underlying world state captured in the camera's eye, with an overarching goal of general computational image understanding. To date much vision research has approached image understanding by focusing on object detection, only one perspective on the image understanding problem. This project looks at an additional, complimentary way to collect information about the visual world -- by directly analyzing the enormous amount of visually descriptive text on the web to reveal what information is useful to attach to, and extract from pictures. This project presents a comprehensive research program geared toward modeling and exploiting the complimentary nature of words and pictures. One main goal is studying the connection between text and images to learn about depiction -- communication of meaning through pictures. This goal is addressed through 3 broad challenges: 1) Developing a richer vocabulary to describe the information provided by depiction. 2) Developing image representations that can visually capture this more nuanced vocabulary. 3) Constructing a comprehensive joint words and pictures framework. This project has direct significance to many concrete tasks that access images on the internet including: image search, browsing, and organization, as well as commercial applications such as product search, and societally important applications such as web assistance for the blind. Additionally, outputs of this project, including progress toward a natural vocabulary and structure for visual description, have great potential for cross-cutting impact in both the computer vision and natural language communities.

图片直接向观众传达了对世界的视觉描述。计算机视觉致力于设计算法来提取相机眼中捕获的底层世界状态，其总体目标是一般计算图像理解。到目前为止，许多视觉研究已经接近图像理解的目标检测集中，只有一个角度的图像理解问题。 这个项目着眼于一个额外的，免费的方式来收集有关视觉世界的信息--通过直接分析网络上大量的视觉描述性文本来揭示哪些信息是有用的，并从图片中提取。这个项目提出了一个全面的研究计划，面向建模和利用文字和图片的互补性。一个主要目标是研究文本和图像之间的联系，以了解描绘-通过图片传达意义。这个目标是通过三个广泛的挑战来实现的：1）开发更丰富的词汇来描述描述所提供的信息。2)开发图像表示，可以在视觉上捕捉这种更微妙的词汇。3)构建一个全面的联合文字和图片框架。该项目对许多访问互联网上图像的具体任务具有直接意义，包括：图像搜索，浏览和组织，以及商业应用，如产品搜索，以及社会重要应用，如盲人网络援助。此外，该项目的产出，包括在自然词汇和视觉描述结构方面的进展，在计算机视觉和自然语言社区都有很大的交叉影响潜力。