CAREER: Teaching Machines to Recognize Complex Visual Concepts in Images through Compositionality

用于图像识别的现代计算系统可以学会在大量类别中检测物体。然而，为了教会机器识别每一个新的类别，人类操作员需要用分类标签标注大量的图像。在实践中，许多应用程序需要一组自定义的类别。例如，为电子商务应用程序检测不同类型家具的视觉识别模型可能需要非常具体的类别，如“摇椅”、“转椅”、“重点椅”或“重点转椅”。即使是对每种椅子的视觉特征有很好认识的专家领域用户，也必须通过单独注释图像来教系统。该项目的目标是实现更丰富的交互模式，“机器教师”将能够通过对每个新类别重要的视觉特征类型的直接反馈来指导图像识别。为此，我们计划利用组合性原则，在这些原则中，新的类别可以基于更容易识别的基本概念来定义。该项目将把研究与教育结合起来，并让来自代表性不足群体的本科生参与研究。该项目将设计新的模型，通过首先发现并学习识别跨许多类共享的视觉原语来学习组合识别视觉概念。这个过程也将被量身定制，以最大限度地发挥用户可以通过自然交互（包括语言的使用和通过视觉界面的直接操作）来指导模型的环境中的效用。该项目将包括：1)开发从文本描述中组合和交互学习的方法；2)提出自动发现可跨类别组合的原语的方法；3)提出即使在部署之后也能支持交互的模型。这三个研究目标将由一个全面的评估计划、一个在互动环境中展示我们的方法的公共平台和扩大参与活动来补充。这项研究将带来视觉识别模型的新设计，为人们提供更有表现力的方式来指导和训练他们。该奖项反映了美国国家科学基金会的法定使命，并通过使用基金会的知识价值和更广泛的影响审查标准进行评估，被认为值得支持。

Modern computational systems for image recognition can be taught to detect objects among large sets of categories. However, in order to teach machines to recognize every new category, human operators need to annotate a large number of images with categorical labels. In practice many applications require a custom set of categories. For instance, a visual recognition model for detecting different types of furniture for an e-commerce application might require very specific categories such as ‘rocking chair’, ‘swivel chair’, ‘accent chair’, or ‘swivel accent chair’. Even an expert domain user that has a good idea in mind for what should be the visual characteristics that are important to recognize in each type of chair, would have to teach the system through annotating images individually. The goal of this project is to enable richer modes of interaction where ‘machine teachers’ would be able to guide the image recognition through direct feedback on the types of visual characteristics that are important for each new category. To this end we plan to exploit principles of compositionality where new categories can be defined based on basic concepts that are easier to recognize. The project will integrate research with the education and involve undergraduate students from underrepresented groups in the research.This project will devise new models that learn to recognize visual concepts compositionally by first discovering and then learning to recognize visual primitives that are shared across many classes. This process will also be tailored to maximize the utility in an environment where a user can guide the model through natural interactions including the use of language and direct manipulation through a visual interface. The project will be 1) developing methods to compositionally and interactively learn from textual descriptions 2) proposing methods to automatically discover primitives that are composable across categories, and 3) proposing models that can support interactions even after deployment. These three research aims will be complemented by a comprehensive evaluation plan, a public platform that exposes our methods in an interactive environment, and broadening participation activities. This research effort will bring novel designs in visual recognition models that offer people more expressive ways for guiding them and training them.This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.