RI: Medium: Collaborative Research: Text-to-Image Reference Resolution for Image Understanding and Manipulation

RI：媒介：协作研究：用于图像理解和操作的文本到图像参考分辨率

• 批准号: 1563727
• 负责人: Svetlana Lazebnik
• 金额: $ 55万
• 依托单位: University of Illinois at Urbana-Champaign
• 依托单位国家: 美国
• 项目类别: Continuing Grant
• 财政年份: 2016
• 资助国家: 美国
• 起止时间: 2016-06-01 至 2021-05-31
• 项目状态: 已结题
• 来源: https://www.nsf.gov/awardsearch/showAward?AWD_ID=1563727&HistoricalAwards=false
• 关键词: RI; Medium; Collaborative; Research; Text

This project develops new technologies at the interface of computer vision and natural language processing to understand text-to-image relationships. For example, given a captioned image, the project develops techniques which determine which words (e.g. "woman talking on phone", "The farther vehicle") correspond to which image parts. From robotics to human-computer interaction, there are numerous real-world tasks that benefit from practical systems to identify objects in scenes based on language and understand language based on visual context. In particular, the project develops the first language-based image authoring tool which allows users to edit or synthesize realistic imagery using only natural language (e.g. "delete the garbage truck from this photo" or "make an image with three boys chasing a shaggy dog"). Beyond the immediate impact of creating new ways for users to access and author digital images, the broader impacts of this work include three focus areas: the development of new benchmarks for the vision and language communities, outreach and undergraduate research, and leadership in promoting diversity. At the core of the project are new techniques for large-scale text-to-image reference resolution (TIRR) that enable systems to automatically identify the image regions that depict entities described in natural language sentences or commands. These techniques advance image interpretation by enabling systems to perform partial matching between images and sentences, referring expression understanding, and image-based question answering. They also advance image manipulation by enabling systems that can synthesize images starting from a textual description, or modify images based on natural language commands. The main technical contributions of the project are: (1) benchmark datasets for TIRR with comprehensive large-scale gold standard annotations that will make TIRR a standard task for recognition; (2) principled new representations for text-to-image annotations that expose the compositional nature of language using the formalism of the denotation graph; (3) new models for TIRR that perform an explicit alignment (grounding) of words and phrases to image regions guided by the structure of the denotation graph; (4) applications of TIRR methods to referring expression understanding and visual question answering; and (5) applications of TIRR to image creation and manipulation based on natural language input.

该项目开发计算机视觉和自然语言处理界面的新技术，以理解文本与图像之间的关系。例如，给定一个标题图像，该项目开发的技术，以确定哪些词(例如；“打电话的女人”，“远处的车辆”)对应于图像的哪个部分。从机器人到人机交互，有许多现实世界的任务受益于基于语言识别场景中的对象和基于视觉上下文理解语言的实用系统。特别是，该项目开发了第一个基于语言的图像创作工具，允许用户仅使用自然语言编辑或合成逼真的图像。“从这张照片中删除垃圾车”或“制作一个三个男孩追逐一只毛茸茸的狗的照片”)。除了为用户创造获取和创作数字图像的新途径所带来的直接影响外，这项工作的更广泛影响还包括三个重点领域：为视觉和语言社区制定新的基准，拓展和本科生研究，以及在促进多样性方面的领导作用。该项目的核心是大规模文本到图像参考分辨率（TIRR）的新技术，该技术使系统能够自动识别用自然语言句子或命令描述的描述实体的图像区域。这些技术通过使系统能够在图像和句子之间执行部分匹配、参考表达理解和基于图像的问答来推进图像解释。它们还推动了图像处理，使系统能够从文本描述开始合成图像，或基于自然语言命令修改图像。项目的主要技术贡献有：(1)具有全面大规模金标准注释的TIRR基准数据集，使TIRR成为识别的标准任务；(2)文本到图像注释的原则性新表示，使用外延图的形式主义揭示语言的组合性质；(3)新的TIRR模型，在表示图结构的引导下，将单词和短语显式对齐（接地）到图像区域；(4) TIRR方法在参考表情理解和视觉问答中的应用；(5) TIRR在基于自然语言输入的图像创建和处理中的应用。