CAREER: Exploiting Deep Generative Models for Visual Recognition

现代视觉识别系统已经在标准基准上取得了令人印象深刻的结果，并在给定大量数据和注释的情况下对常见对象和场景可靠地工作。不幸的是，目前的系统很难检测到稀有或看不见的物体，也无法适应新的领域。研究人员、工程师和/或领域专家必须捕获和注释大量真实数据，这些数据对于普通对象来说是昂贵的，而对于稀有对象和角落情况(即当多个独特条件同时发生时发生的情况)则是不切实际的。为了应对上述挑战，并自动创建和标记完全描述拐角情况的数据，该项目利用了大型生成性模型丰富的组成结构和强大的合成能力。通过使用这些模型，可以快速合成具有未知视觉元素(例如，新姿势、天气、照明等)的不同对象和场景。该项目将开发识别算法，可以识别稀有/看不见的物体，以适应不断变化的环境。该项目有可能对各种应用产生变革，如自动驾驶、辅助机器人、医疗保健、电子商务和混合现实。此外，这项研究将转化为代码、模型、课程和教程，供不同的利益相关者广泛访问，以及与更广泛社区接触的教育和研究计划。直接使用生成模型是具有挑战性的，因为随机采样的图像不太可能覆盖可以改进识别系统的角例。为了合成更接近长尾分布和新领域的数据，这个项目将专注于三个研究推动力。首先，该项目通过探索自动生成数据和注释的不同方法，通过生成模型解决学习视觉识别问题。其次，该项目将通过生成模型来分析视觉识别系统，方法是合成各种不断演变的测试数据，以询问系统并理解偏差。最后，该项目将自动选择和调整生成性模型以适应新的领域和任务。这三个推力是紧密相连的，因为一旦算法识别出不符合我们当前系统的硬例子，这些例子就可以用来闭合训练和分析之间的循环。最后，研究人员将通过比较使用或不使用生成模型的方法来评估开发的方法。这一奖项反映了NSF的法定使命，并通过使用基金会的智力优势和更广泛的影响审查标准进行评估，被认为值得支持。

Modern visual recognition systems have achieved impressive results on standard benchmarks and work reliably for common objects and scenes, given massive data and annotations. Unfortunately, current systems struggle to detect rare or unseen objects and fail to adapt to new domains. Researchers, engineers and/or domain experts have to capture and annotate huge amounts of real data, which are costly for common objects and impractical for rare objects and corner cases (i.e., cases that occur when multiple unique conditions simultaneously occur). To address the above challenges and automatically create and label data that fully depict the corner cases, this project leverages the rich compositional structure and powerful synthesis capacity of large-scale generative models. By using these models that can quickly synthesize diverse objects and scenes with an unknown visual elements (e.g., new poses, weather, lighting, etc.). This project will develop recognition algorithms that can recognize rare/unseen objects to adapt to continuously changing environments. This project has a potential to be transformative for various applications, such as autonomous driving, assistive robots, healthcare, e-commerce, and mixed reality. Furthermore, this research will translate to code, models, courses, and tutorials, that are widely accessible to diverse stakeholders and education and research programs that engage with the broader community. Directly using generative models is challenging, as it is highly unlikely that a randomly sampled image will cover a corner case that can improve recognition systems. To synthesize data that more closely resemble the long-tail distribution and new domains, this project will focus on three research thrusts. First, the project addresses learning visual recognition via generative models by exploring different methods of automatically generating data and annotations. Second, the project will analyze visual recognition systems through generative models by synthesizing diverse, continuously evolving test data to interrogate the system and understand the biases. Finally, the project will automatically select and adapt generative models to new domains and tasks. These three thrusts are tightly connected, as once the algorithms identify hard examples that fail our current system, these examples can be used to close the loop between training and analysis. Finally, investigators will evaluate the developed method by comparing methods with or without using generative models.This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.