RI: Small: Texture2Text: Rich Language-Based Understanding of Textures for Recognition and Synthesis

这个项目开发了视觉和自然语言界面的技术，以理解和合成纹理。例如，在给定纹理的情况下，该项目开发了提供对图案的描述的技术(例如，“表面光滑”、“白色背景上的红色圆点”)。纹理的语义理解技术有利于大量的应用，从机器人学(理解表面的材料属性是交互的关键)到分析各种形式的图像，用于气象学、海洋学、保护学、地质学和林业。此外，该项目还开发了允许根据自然语言描述修改和合成纹理的技术(例如，“使墙纸更加之字形”、“创建蜂窝图案”)，从而实现了创建纹理的新的以人为中心的工具。除了这个项目带来的众多应用外，这项工作的更广泛的影响包括：为计算机视觉和语言社区开发新的基准和软件，本科生研究和推广，以及与研究人员和公民科学家在保护领域的合作。这项研究将视觉纹理映射到自然语言描述，反之亦然。这项研究通过提供对真实成像条件、自然场景中的杂乱和遮挡具有健壮性的纹理表示来推进计算机视觉；通过提供使用描述来搜索和检索纹理的新方法来进行内容检索；以及通过提供使用描述来创建和修改纹理的新方法来进行图像处理。该项目的主要技术贡献是：(1)将纹理模型的各个方面与深度学习相结合的原则性体系结构，以实现对纹理表示的端到端学习；(2)通过可视化来理解这些表示的属性的技术；(3)评估基于语言的纹理理解技术的大规模基准；(4)纹理字幕的新模型；(5)纹理表示在细粒度识别和语义分割方面的应用；以及(6)使用自然语言描述检索和创建纹理的技术。

This project develops techniques at the interface of vision and natural language to understand and synthesize textures. For example, given a texture the project develops techniques that provide a description of the pattern (e.g., "the surface is slippery", "red polka-dots on a white background"). Techniques for semantic understanding of textures benefit a large number of applications ranging from robotics where understanding material properties of surfaces is key to interaction, to analysis of various forms of imagery for meteorology, oceanography, conservation, geology, and forestry. In addition, the project develops techniques that allow modification and synthesis of textures based on natural language descriptions (e.g., "make the wallpaper more zig-zagged", "create a honeycombed pattern"), enabling new human-centric tools for creating textures. In addition to the numerous applications enabled by this project, the broader impacts of the work include: the development of new benchmarks and software for computer vision and language communities, undergraduate research and outreach, and collaboration with researchers and citizen scientists in areas of conservation.This research maps visual textures to natural language descriptions and vice versa. The research advances computer vision by providing texture representations that are robust to realistic imaging conditions, clutter, and occlusions in natural scenes; content retrieval by providing new ways to search and retrieve textures using descriptions; and image manipulation by providing new ways to create and modify textures using descriptions. The main technical contributions of the project are: (1) principled architectures that combine aspects of texture models with deep learning to enable end-to-end learning of texture representations; (2) techniques for understanding the properties of these representations through visualizations; (3) a large-scale benchmark to evaluate techniques for language-based texture understanding; (4) new models for texture captioning; (5) applications of texture representations for fine-grained recognition and semantic segmentation; and (6) techniques for retrieving and creating textures using natural language descriptions.