CAREER: Teaching Machines to Recognize Complex Visual Concepts in Images through Compositionality

用于图像识别的现代计算系统可以被教导在大的类别集合中检测对象。然而，为了教会机器识别每一个新的类别，人类操作员需要用分类标签注释大量图像。实际上，许多应用程序都需要一组自定义的类别。例如，用于检测电子商务应用的不同类型的家具的视觉识别模型可能需要非常特定的类别，诸如“摇椅”、“转椅”、“口音椅”或“转椅口音椅”。即使是一个专家领域的用户，有一个好主意，什么应该是重要的视觉特征，以识别每种类型的椅子，将不得不教系统通过注释图像单独。该项目的目标是实现更丰富的交互模式，其中“机器教师”将能够通过对每个新类别重要的视觉特征类型的直接反馈来指导图像识别。为此，我们计划利用组合性的原则，新的类别可以根据更容易识别的基本概念定义。该项目将把研究与教育结合起来，并让来自代表性不足群体的本科生参与研究。该项目将设计新的模型，通过首先发现并学习识别跨多个类共享的视觉基元来学习识别视觉概念。这一过程也将被定制，以最大限度地提高效用的环境中，用户可以引导模型通过自然的互动，包括使用语言和直接操纵通过视觉界面。该项目将1）开发从文本描述中进行组合和交互式学习的方法2）提出自动发现跨类别可组合的原语的方法，以及3）提出即使在部署后也可以支持交互的模型。这三个研究目标将得到一个全面的评估计划，一个在互动环境中展示我们的方法的公共平台，以及扩大参与活动的补充。该研究成果将为视觉识别模型带来新颖的设计，为人们提供更具表现力的指导和培训方式。该奖项反映了NSF的法定使命，并通过使用基金会的智力价值和更广泛的影响审查标准进行评估，被认为值得支持。

Modern computational systems for image recognition can be taught to detect objects among large sets of categories. However, in order to teach machines to recognize every new category, human operators need to annotate a large number of images with categorical labels. In practice many applications require a custom set of categories. For instance, a visual recognition model for detecting different types of furniture for an e-commerce application might require very specific categories such as ‘rocking chair’, ‘swivel chair’, ‘accent chair’, or ‘swivel accent chair’. Even an expert domain user that has a good idea in mind for what should be the visual characteristics that are important to recognize in each type of chair, would have to teach the system through annotating images individually. The goal of this project is to enable richer modes of interaction where ‘machine teachers’ would be able to guide the image recognition through direct feedback on the types of visual characteristics that are important for each new category. To this end we plan to exploit principles of compositionality where new categories can be defined based on basic concepts that are easier to recognize. The project will integrate research with the education and involve undergraduate students from underrepresented groups in the research.This project will devise new models that learn to recognize visual concepts compositionally by first discovering and then learning to recognize visual primitives that are shared across many classes. This process will also be tailored to maximize the utility in an environment where a user can guide the model through natural interactions including the use of language and direct manipulation through a visual interface. The project will be 1) developing methods to compositionally and interactively learn from textual descriptions 2) proposing methods to automatically discover primitives that are composable across categories, and 3) proposing models that can support interactions even after deployment. These three research aims will be complemented by a comprehensive evaluation plan, a public platform that exposes our methods in an interactive environment, and broadening participation activities. This research effort will bring novel designs in visual recognition models that offer people more expressive ways for guiding them and training them.This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.