CRII: RI: Multi-Source Domain Generalization Approaches to Visual Attribute Detection

该项目研究如何准确而稳健地从图像（视频和3D数据）中检测属性，目的是开发和公开提供有效的属性检测工具。视觉属性是指来自对象、场景和活动（例如，四条腿的、户外的和拥挤的）的视觉内容的人类可命名和机器可检测的固有特征。它们具有多用途的特性和应用潜力，提供了一个自然的人机交互通道，使人类参与机器视觉算法的循环，作为一个人组成类别和描述实例的基本构建块，并为统计学习模型带来丰富的先验知识和正则化，等等。该项目推进了利用属性进行各种视觉识别和搜索任务的长期追求。该项目还积极吸引研究生和本科生，并向当地高中生伸出援手。该项目的研究成果可以影响几个相关的社区，如NLP，语音和机器人等。这项研究明确地解决了属性检测器应该很好地泛化不同类别的需求，包括那些以前未见过的类别。研究小组采用多源域泛化的方法，将每个类别作为一个域。特别是，这个项目开发了新的特征提取工具，专门用于中级属性，而不是主要为高级视觉识别设计和测试的传统特征。该项目包括三个主要的推力，围绕属性检测和领域泛化之间的类比的关键动机。它首先学习细粒度的“浅”特征映射（Thrust I），以提取类别不变的属性判别信号，然后“更深入”地研究特征提取框架——Fisher向量（Thrust II）和卷积神经网络（Thrust III）——以修改它们以进行属性检测。

This project investigates how to accurately and robustly detect attributes from images (videos, and 3D data), with the goal of developing and publicly providing effective attribute detection tools. Visual attributes refer to human-namable and machine-detectable inherent characteristics of visual content from objects, scenes, and activities (e.g., four-legged, outdoor, and crowded). They possess versatile properties and application potentials by offering a natural human-computer interaction channel for involving humans in the loop of machine vision algorithms, serving as basic building blocks for one to compose categories and describe instances, and bringing rich prior knowledge and regularization to statistical learning models, to name a few. The project advances the long-standing pursuit of utilizing attributes for a wide variety of visual recognition and search tasks. The project also actively engages graduate and undergraduate students, and outreaches local high-school students. The research results from this project can impact several related communities such as NLP, speech, and robotics, etc.. This research explicitly tackles the need that attribute detectors should generalize well across different categories, including those previously unseen ones. The research team approaches the problem based on multi-source domain generalization by taking each category as a domain. In particular, this project develops new feature extraction tools tailored to account for the middle-level attributes, as opposed to the traditional features primarily designed and tested for high-level visual recognition. The project consists of three major thrusts hinging on the key motivation of the analogy between attribute detection and domain generalization. It begins by learning a fine-grained "shallow" feature mapping (Thrust I) to distill attribute-discriminative signals that are category-invariant, and then investigates "deeper" into the feature extraction frameworks - Fisher vectors (Thrust II) and convolutional neural networks (Thrust III)-to revise them for the purpose of attribute detection.