RI: Medium: Collaborative Research: Text-to-Image Reference Resolution for Image Understanding and Manipulation

该项目在计算机视觉和自然语言处理的界面上开发新技术，以理解文本到图像的关系。例如，给定一个标题图像，该项目开发的技术，确定哪些词（例如“女人打电话”，“更远的车辆”）对应于图像的哪些部分。从机器人技术到人机交互，有许多现实世界的任务受益于实用系统，可以根据语言识别场景中的对象，并根据视觉上下文理解语言。特别是，该项目开发了第一个基于语言的图像创作工具，允许用户仅使用自然语言编辑或合成逼真的图像（例如“从这张照片中删除垃圾车”或“制作一个三个男孩追逐毛茸茸的狗的图像”）。除了为用户创建访问和创作数字图像的新方法的直接影响外，这项工作的更广泛影响包括三个重点领域：为视觉和语言社区制定新的基准，外联和本科生研究，以及促进多样性的领导力。该项目的核心是大规模文本到图像参考分辨率（TIRR）的新技术，使系统能够自动识别描述自然语言句子或命令中描述的实体的图像区域。这些技术通过使系统能够执行图像和句子之间的部分匹配、引用表达理解和基于图像的问题回答来推进图像解释。它们还通过使系统能够从文本描述开始合成图像或基于自然语言命令修改图像来推进图像操作。该项目的主要技术贡献是：（1）TIRR的基准数据集，具有全面的大规模黄金标准注释，将使TIRR成为识别的标准任务;（2）文本到图像注释的原则性新表示，使用表示图的形式主义揭示语言的组成性质;（3）执行显式对齐的TIRR的新模型（4）TIRR方法在指称表达理解和视觉问答中的应用;以及（5）TIRR在基于自然语言输入的图像创建和处理中的应用。

This project develops new technologies at the interface of computer vision and natural language processing to understand text-to-image relationships. For example, given a captioned image, the project develops techniques which determine which words (e.g. "woman talking on phone", "The farther vehicle") correspond to which image parts. From robotics to human-computer interaction, there are numerous real-world tasks that benefit from practical systems to identify objects in scenes based on language and understand language based on visual context. In particular, the project develops the first language-based image authoring tool which allows users to edit or synthesize realistic imagery using only natural language (e.g. "delete the garbage truck from this photo" or "make an image with three boys chasing a shaggy dog"). Beyond the immediate impact of creating new ways for users to access and author digital images, the broader impacts of this work include three focus areas: the development of new benchmarks for the vision and language communities, outreach and undergraduate research, and leadership in promoting diversity. At the core of the project are new techniques for large-scale text-to-image reference resolution (TIRR) that enable systems to automatically identify the image regions that depict entities described in natural language sentences or commands. These techniques advance image interpretation by enabling systems to perform partial matching between images and sentences, referring expression understanding, and image-based question answering. They also advance image manipulation by enabling systems that can synthesize images starting from a textual description, or modify images based on natural language commands. The main technical contributions of the project are: (1) benchmark datasets for TIRR with comprehensive large-scale gold standard annotations that will make TIRR a standard task for recognition; (2) principled new representations for text-to-image annotations that expose the compositional nature of language using the formalism of the denotation graph; (3) new models for TIRR that perform an explicit alignment (grounding) of words and phrases to image regions guided by the structure of the denotation graph; (4) applications of TIRR methods to referring expression understanding and visual question answering; and (5) applications of TIRR to image creation and manipulation based on natural language input.