RI: Medium: Collaborative Research: Semantically Discriminative : Guiding Mid-Level Representations for Visual Object Recognition with External Knowledge

这个项目探索了(半)自动的方法，通过在学习区分对象的统计学习过程中引入对象属性的外部知识，为视觉对象分类创建“语义上有区别的”中层线索。具体地说，PI研究四个关键思想：(1)利用对象类别上的分类法来通知特征选择算法，以便它们集中在给定粒度的标签预测的最抽象描述上；(2)利用由相同分类法传达的对象间关系来指导上下文学习，使得它捕获的不仅仅是简单的数据驱动的共现；(3)探索从自然语言中提取的视觉属性的效用，既作为辅助学习问题来偏向对象分类模型，也作为必须使用非传统的人类监督策略梳理出的序数属性；(4)挖掘既独特又可命名的属性，超越了人工构建的语义。该项目在计算机视觉和机器学习方面都做出了原创性的贡献，是迈向基于语义的对象分类的不可或缺的一步。虽然主流方法将人类知识简化为样本上的单纯类别标签，但这项工作在学习管道中更深入、更早地利用了语义丰富的知识。该方法导致视觉系统不太容易过度匹配附带的视觉模式，以及容易扩展到新的视觉学习任务的表示。在研究界之外，这项工作通过对研究生和本科生进行跨学科培训，以及通过讲习班和夏令营向大学前教育工作者和学生推广，鼓励年轻学生攻读科学和工程，产生了更广泛的影响。

This project explores (semi-)automatic ways to create "semantically discriminative" mid-level cues for visual object categorization, by introducing external knowledge of object properties into the statistical learning procedures that learn to distinguish them. In particular, the PIs investigate four key ideas: (1) exploiting taxonomies over object categories to inform feature selection algorithms such that they home in on the most abstract description for a given granularity of label predictions; (2) leveraging inter-object relationships conveyed by the same taxonomies to guide context learning, so that it captures more than simple data-driven co-occurrences; (3) exploring the utility of visual attributes drawn from natural language, both as auxiliary learning problems to bias models for object categorization, as well as ordinal properties that must be teased out using non-traditional human supervision strategies; (4) mining attributes that are both distinctive and human-nameable, moving beyond manually constructed semantics.The project entails original contributions in both computer vision and machine learning, and is an integral step towards semantically-grounded object categorization. Whereas mainstream approaches reduce human knowledge to mere category labels on exemplars, this work leverages semantically rich knowledge more deeply and earlier in the learning pipeline. The approach results in vision systems that are less prone to overfit incidental visual patterns, and representations that are readily extendible to novel visual learning tasks. Beyond the research community, the work has broader impact through inter-disciplinary training of graduate and undergraduate students, and outreach to pre-college educators and students through workshops and summer camps encouraging young students to pursue science and engineering.