RI: Medium: Collaborative Research: Write A Classifier: Learning Fine-Grained Visual Classifiers from Text and Images

该项目开发了使用文本叙事和图像的学习策略，使学习有效，而不需要典型的视觉学习算法学习类边界所需的大量图像。研究小组研究了从图像和细粒度分类的文本描述中联合学习视觉概念的计算模型，例如，区分鸟类。研究活动在计算机视觉、自然语言处理和机器学习三个领域产生了广泛的影响。开发自动理解图像和视频内容的算法是一个巨大的需求，在网络搜索、图像和视频存档和检索、监控应用、机器人导航等方面有许多潜在的应用。开发一个可以使用叙事来定义和识别类别的智能系统有各种各样的应用。本项目解决了两个研究问题：第一，给定特定领域的视觉语料库和文本语料库，如何共同有效地学习视觉概念？第二，考虑到这两种模式，如何仅使用领域中新类别的纯文本描述来促进学习新的视觉概念？研究团队从三个方面着手解决这个问题：学习、自然语言处理（NLP）和计算机视觉。在学习方面，该项目研究和开发了适合学习和预测带有侧文本信息的视觉分类器的算法。在自然语言处理方面，该项目旨在开发新的方法，通过人类计算和视觉信号的反馈，从文本中学习全局和局部判别类别级属性及其值。该项目研究了用于视觉文本检测的监督和无监督方法，以及用于深度语言理解的学习方法，以从噪声视觉文本中构建丰富的领域模型。在视觉方面，该项目解决了使用侧文本信息进行检测和分类的任务。该项目研究了一般类别的形状和外观模型，可以专门用于不同的下属，以一种允许在适当的几何环境中解释文本信息的方式，并处理观点和表达的可变性。

This project develops the learning strategy using textual narrative and images makes the learning effective without a huge number of images that a typical visual learning algorithm would need to learn the class boundaries. The research team investigates computational models for joint learning of visual concepts from images and textual descriptions of fine-grained categories, for example, discriminating between bird species. The research activities have broader impact in three fields: computer vision, natural language processing, and machine learning. There is a huge need to develop algorithms to automatically understand the content of images and videos, with numerous potential applications in web searches, image and video archival and retrieval, surveillance applications, robot navigation and others. There are various applications for developing an intelligent system that can use narrative to define and recognize categories.This project addresses two research questions: First, given a visual corpus and a textual corpus about a specific domain, how to jointly and effectively learn visual concepts? Second, given these two modalities how to facilitate learning novel visual concepts using only pure textual descriptions of novel categories in the domain? The research team approaches the problem on three integrated fronts: Learning, Natural Language Processing (NLP), and Computer Vision. On the learning front, the project investigates and develops algorithms suitable for learning and predicting visual classifiers with side textual information. On the NLP front, the project aims to develop novel methods for learning global and local discriminative category-level attributes and their values from text, with feedback from human computation and visual signal. The project investigates supervised and unsupervised methods for detecting visual text, and learning methods for deep language understanding to build such rich domain models from the noisy visual text. On the Vision front, the project addresses the tasks of detection and classification with side textual information. The project investigates models for the shape and appearance of a general category that can specialize to different subordinates, in a way that allows interpreting information from text within a proper geometric context, and handle variability in viewpoints and articulation.