RI: Medium: Collaborative Research: Write A Classifier: Learning Fine-Grained Visual Classifiers from Text and Images

RI：媒介：协作研究：编写分类器：从文本和图像中学习细粒度视觉分类器

• 批准号: 1409683
• 负责人: Ahmed Elgammal
• 金额: $ 50万
• 依托单位: Rutgers University New Brunswick
• 依托单位国家: 美国
• 项目类别: Continuing Grant
• 财政年份: 2014
• 资助国家: 美国
• 起止时间: 2014-06-15 至 2021-05-31
• 项目状态: 已结题
• 来源: https://www.nsf.gov/awardsearch/showAward?AWD_ID=1409683&HistoricalAwards=false
• 关键词: RI; Medium; Collaborative; Research; Write

This project develops the learning strategy using textual narrative and images makes the learning effective without a huge number of images that a typical visual learning algorithm would need to learn the class boundaries. The research team investigates computational models for joint learning of visual concepts from images and textual descriptions of fine-grained categories, for example, discriminating between bird species. The research activities have broader impact in three fields: computer vision, natural language processing, and machine learning. There is a huge need to develop algorithms to automatically understand the content of images and videos, with numerous potential applications in web searches, image and video archival and retrieval, surveillance applications, robot navigation and others. There are various applications for developing an intelligent system that can use narrative to define and recognize categories.This project addresses two research questions: First, given a visual corpus and a textual corpus about a specific domain, how to jointly and effectively learn visual concepts? Second, given these two modalities how to facilitate learning novel visual concepts using only pure textual descriptions of novel categories in the domain? The research team approaches the problem on three integrated fronts: Learning, Natural Language Processing (NLP), and Computer Vision. On the learning front, the project investigates and develops algorithms suitable for learning and predicting visual classifiers with side textual information. On the NLP front, the project aims to develop novel methods for learning global and local discriminative category-level attributes and their values from text, with feedback from human computation and visual signal. The project investigates supervised and unsupervised methods for detecting visual text, and learning methods for deep language understanding to build such rich domain models from the noisy visual text. On the Vision front, the project addresses the tasks of detection and classification with side textual information. The project investigates models for the shape and appearance of a general category that can specialize to different subordinates, in a way that allows interpreting information from text within a proper geometric context, and handle variability in viewpoints and articulation.

该项目开发了使用文本叙事和图像的学习策略，使学习有效，而无需大量的图像，典型的视觉学习算法需要学习类边界。 研究小组调查了用于从图像和细粒度类别的文本描述中联合学习视觉概念的计算模型，例如，区分鸟类。 研究活动在三个领域产生了更广泛的影响：计算机视觉，自然语言处理和机器学习。有一个巨大的需求，以开发算法来自动理解图像和视频的内容，在网络搜索，图像和视频存档和检索，监控应用，机器人导航和其他许多潜在的应用。有各种各样的应用程序来开发一个智能系统，可以使用叙事来定义和识别categories.本项目解决两个研究问题：第一，给定一个特定领域的视觉语料库和文本语料库，如何联合和有效地学习视觉概念？第二，给定这两种方式，如何促进学习新的视觉概念，只使用纯文本描述的新类别的领域？研究团队从三个方面着手解决这个问题：学习、自然语言处理（NLP）和计算机视觉。在学习方面，该项目研究和开发适合学习和预测具有侧文本信息的视觉分类器的算法。在NLP方面，该项目旨在开发新的方法，用于从文本中学习全局和局部判别类别级属性及其值，并从人类计算和视觉信号中获得反馈。该项目研究了用于检测视觉文本的有监督和无监督方法，以及用于深度语言理解的学习方法，以从嘈杂的视觉文本中构建丰富的领域模型。在视觉方面，该项目解决了检测和分类与侧文本信息的任务。该项目研究了一个一般类别的形状和外观模型，该类别可以专门针对不同的下属，以一种允许在适当的几何背景下解释文本信息的方式，并处理视点和关节的变化。