RI: Small: Texture2Text: Rich Language-Based Understanding of Textures for Recognition and Synthesis

该项目开发了视觉和自然语言界面的技术来理解和合成纹理。例如，给定一个纹理，该项目开发了提供图案描述的技术（例如，“表面光滑”，“白色背景上的红色圆点”）。纹理的语义理解技术有利于大量的应用，从理解表面材料特性的机器人到分析气象学、海洋学、自然保护、地质学和林业的各种形式的图像。此外，该项目还开发了基于自然语言描述的纹理修改和合成技术（例如，“使墙纸更加锯齿状”，“创建蜂巢状图案”），从而为创建纹理提供了新的以人为中心的工具。除了这个项目带来的众多应用之外，这项工作的更广泛影响还包括：为计算机视觉和语言社区开发新的基准和软件，本科研究和推广，以及与保护领域的研究人员和公民科学家的合作。这项研究将视觉纹理映射到自然语言描述，反之亦然。该研究通过提供对自然场景中的真实成像条件、杂波和遮挡具有鲁棒性的纹理表示来推进计算机视觉；内容检索：提供使用描述搜索和检索纹理的新方法；通过提供使用描述创建和修改纹理的新方法来处理图像。该项目的主要技术贡献有：(1)将纹理模型的各个方面与深度学习相结合的原则性架构，以实现纹理表示的端到端学习；(2)通过可视化来理解这些表示的属性的技术；(3)评估基于语言的纹理理解技术的大规模基准；(4)纹理标注新模型；(5)纹理表示在细粒度识别和语义分割中的应用；(6)使用自然语言描述检索和创建纹理的技术。

This project develops techniques at the interface of vision and natural language to understand and synthesize textures. For example, given a texture the project develops techniques that provide a description of the pattern (e.g., "the surface is slippery", "red polka-dots on a white background"). Techniques for semantic understanding of textures benefit a large number of applications ranging from robotics where understanding material properties of surfaces is key to interaction, to analysis of various forms of imagery for meteorology, oceanography, conservation, geology, and forestry. In addition, the project develops techniques that allow modification and synthesis of textures based on natural language descriptions (e.g., "make the wallpaper more zig-zagged", "create a honeycombed pattern"), enabling new human-centric tools for creating textures. In addition to the numerous applications enabled by this project, the broader impacts of the work include: the development of new benchmarks and software for computer vision and language communities, undergraduate research and outreach, and collaboration with researchers and citizen scientists in areas of conservation.This research maps visual textures to natural language descriptions and vice versa. The research advances computer vision by providing texture representations that are robust to realistic imaging conditions, clutter, and occlusions in natural scenes; content retrieval by providing new ways to search and retrieve textures using descriptions; and image manipulation by providing new ways to create and modify textures using descriptions. The main technical contributions of the project are: (1) principled architectures that combine aspects of texture models with deep learning to enable end-to-end learning of texture representations; (2) techniques for understanding the properties of these representations through visualizations; (3) a large-scale benchmark to evaluate techniques for language-based texture understanding; (4) new models for texture captioning; (5) applications of texture representations for fine-grained recognition and semantic segmentation; and (6) techniques for retrieving and creating textures using natural language descriptions.