CAREER: Teaching Machines to Recognize Complex Visual Concepts in Images through Compositionality

用于图像识别的现代计算系统可以被教导来从大的类别集中检测对象。然而，为了教会机器识别每一个新的类别，人类操作员需要用分类标签来标注大量的图像。实际上，许多应用程序都需要一组定制的类别。例如，用于检测电子商务应用程序中不同类型家具的视觉识别模型可能需要非常特定的类别，如“摇椅”、“转椅”、“转椅”或“转椅”。即使是一个专家领域的用户，在脑海中也有一个很好的想法，知道在每种类型的椅子上应该识别哪些视觉特征是重要的，也必须通过单独注释图像来教授系统。该项目的目标是实现更丰富的交互模式，在这种模式下，机器教师将能够通过直接反馈对每个新类别重要的视觉特征类型来指导图像识别。为此，我们计划利用组合性原则，在这种原则下，可以根据更容易识别的基本概念来定义新的类别。该项目将把研究与教育结合起来，并让来自代表性不足群体的本科生参与研究。该项目将设计出新的模型，通过首先发现然后学习识别在许多班级共享的视觉原语来学习识别组成的视觉概念。这一过程还将被量身定做，以便在用户可以通过自然交互(包括使用语言和通过可视界面直接操纵)指导模型的环境中最大限度地发挥效用。该项目将是1)开发方法，以组合和交互地学习文本描述，2)提出方法，以自动发现可跨类别组合的原语，以及3)提出模型，即使在部署后也能支持交互。这三个研究目标将得到一个全面的评估计划、一个在互动环境中展示我们的方法的公共平台以及扩大参与活动的补充。这项研究工作将在视觉识别模型中带来新颖的设计，为人们提供更具表现力的方式来指导和培训他们。该奖项反映了NSF的法定使命，并通过使用基金会的智力优势和更广泛的影响审查标准进行评估，被认为值得支持。

Modern computational systems for image recognition can be taught to detect objects among large sets of categories. However, in order to teach machines to recognize every new category, human operators need to annotate a large number of images with categorical labels. In practice many applications require a custom set of categories. For instance, a visual recognition model for detecting different types of furniture for an e-commerce application might require very specific categories such as ‘rocking chair’, ‘swivel chair’, ‘accent chair’, or ‘swivel accent chair’. Even an expert domain user that has a good idea in mind for what should be the visual characteristics that are important to recognize in each type of chair, would have to teach the system through annotating images individually. The goal of this project is to enable richer modes of interaction where ‘machine teachers’ would be able to guide the image recognition through direct feedback on the types of visual characteristics that are important for each new category. To this end we plan to exploit principles of compositionality where new categories can be defined based on basic concepts that are easier to recognize. The project will integrate research with the education and involve undergraduate students from underrepresented groups in the research.This project will devise new models that learn to recognize visual concepts compositionally by first discovering and then learning to recognize visual primitives that are shared across many classes. This process will also be tailored to maximize the utility in an environment where a user can guide the model through natural interactions including the use of language and direct manipulation through a visual interface. The project will be 1) developing methods to compositionally and interactively learn from textual descriptions 2) proposing methods to automatically discover primitives that are composable across categories, and 3) proposing models that can support interactions even after deployment. These three research aims will be complemented by a comprehensive evaluation plan, a public platform that exposes our methods in an interactive environment, and broadening participation activities. This research effort will bring novel designs in visual recognition models that offer people more expressive ways for guiding them and training them.This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.