RI: Medium: Collaborative Research: Text-to-Image Reference Resolution for Image Understanding and Manipulation

这个项目开发了计算机视觉和自然语言处理接口的新技术，以理解文本与图像的关系。例如，给定一张带字幕的图片，该项目开发了一些技术，来确定哪些单词(如“女人在电话上说话”、“远处的车辆”)对应于图像的哪些部分。从机器人到人机交互，有许多现实世界的任务受益于实际系统，基于语言识别场景中的对象，并基于视觉上下文理解语言。特别是，该项目开发了第一个基于语言的图像创作工具，允许用户仅使用自然语言编辑或合成逼真的图像(例如，“从这张照片中删除垃圾车”或“制作三个男孩追逐一只毛茸茸的狗的图像”)。除了为用户创造获取和创作数字图像的新方式的直接影响外，这项工作的更广泛的影响包括三个重点领域：为视觉和语言社区制定新的基准、外联和本科生研究以及在促进多样性方面发挥领导作用。该项目的核心是大规模文本到图像参考分辨率(TIRR)的新技术，使系统能够自动识别描述以自然语言句子或命令描述的实体的图像区域。这些技术通过使系统能够执行图像和句子之间的部分匹配、参考表达理解和基于图像的问题回答来推进图像解释。它们还通过使系统能够从文本描述开始合成图像或基于自然语言命令修改图像来推进图像处理。该项目的主要技术贡献是：(1)TIRR的基准数据集，具有全面的大规模黄金标准注释，将使TIRR成为识别的标准任务；(2)原则性的文本到图像注释的新表示法，使用表示图的形式来揭示语言的组成性质；(3)TIRR的新模型，其执行单词和短语与图像区域的明确对齐(基础)，以表示图的结构指导；(4)TIRR方法在涉及表情理解和视觉问题回答方面的应用；以及(5)TIRR在基于自然语言输入的图像创建和操作中的应用。

This project develops new technologies at the interface of computer vision and natural language processing to understand text-to-image relationships. For example, given a captioned image, the project develops techniques which determine which words (e.g. "woman talking on phone", "The farther vehicle") correspond to which image parts. From robotics to human-computer interaction, there are numerous real-world tasks that benefit from practical systems to identify objects in scenes based on language and understand language based on visual context. In particular, the project develops the first language-based image authoring tool which allows users to edit or synthesize realistic imagery using only natural language (e.g. "delete the garbage truck from this photo" or "make an image with three boys chasing a shaggy dog"). Beyond the immediate impact of creating new ways for users to access and author digital images, the broader impacts of this work include three focus areas: the development of new benchmarks for the vision and language communities, outreach and undergraduate research, and leadership in promoting diversity. At the core of the project are new techniques for large-scale text-to-image reference resolution (TIRR) that enable systems to automatically identify the image regions that depict entities described in natural language sentences or commands. These techniques advance image interpretation by enabling systems to perform partial matching between images and sentences, referring expression understanding, and image-based question answering. They also advance image manipulation by enabling systems that can synthesize images starting from a textual description, or modify images based on natural language commands. The main technical contributions of the project are: (1) benchmark datasets for TIRR with comprehensive large-scale gold standard annotations that will make TIRR a standard task for recognition; (2) principled new representations for text-to-image annotations that expose the compositional nature of language using the formalism of the denotation graph; (3) new models for TIRR that perform an explicit alignment (grounding) of words and phrases to image regions guided by the structure of the denotation graph; (4) applications of TIRR methods to referring expression understanding and visual question answering; and (5) applications of TIRR to image creation and manipulation based on natural language input.