CAREER: Weakly-Supervised Visual Scene Understanding: Combining Images and Videos, and Going Beyond Semantic Tags

职业：弱监督视觉场景理解：结合图像和视频，超越语义标签

• 批准号: 2150012
• 负责人: Yong Jae Lee
• 金额: $ 50.05万
• 依托单位: University of Wisconsin-Madison
• 依托单位国家: 美国
• 项目类别: Continuing Grant
• 财政年份: 2021
• 资助国家: 美国
• 起止时间: 2021-09-01 至 2024-03-31
• 项目状态: 已结题
• 来源: https://www.nsf.gov/awardsearch/showAward?AWD_ID=2150012&HistoricalAwards=false
• 关键词: CAREER; Weakly; Supervised; Visual; Scene

The internet provides an endless supply of images and videos, replete with weakly-annotated meta-data such as text tags, GPS coordinates, timestamps, or social media sentiments. This huge resource of visual data provides an opportunity to create scalable and powerful recognition algorithms that do not depend on expensive human annotations. The research component of this project develops novel visual scene understanding algorithms that can effectively learn from such weakly-annotated visual data. The main novelty is to combine both images and videos together. The developed algorithms could have broad impact in numerous fields including AI, security, and agricultural sciences. In addition to scientific impact, the project performs complementary educational and outreach activities. Specifically, it provides mentorship to high school, undergraduate, and graduate students, teaches new undergraduate and graduate computer vision courses that have been lacking at UC Davis, and organizes an international workshop on weakly-supervised visual scene understanding.This project develops novel algorithms to advance weakly-supervised visual scene understanding in two complementary ways: (1) learning jointly with both images and videos to take advantage of their complementarity, and (2) learning from weak supervisory signals that go beyond standard semantic tags such as timestamps, captions, and relative comparisons. Specifically, it investigates novel approaches to advance tasks like fully-automatic video object segmentation, weakly-supervised object detection, unsupervised learning of object categories, and mining of localized patterns in the image/video data that are correlated with the weak supervisory signal. Throughout, the project explores ways to understand and mitigate noise in the weak labels and to overcome the domain differences between images and videos.This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.

互联网提供了无穷无尽的图像和视频，充满了缺乏注释的元数据，如文本标签、GPS坐标、时间戳或社交媒体情绪。这种巨大的视觉数据资源为创建可扩展且功能强大的识别算法提供了机会，这些算法不依赖于昂贵的人工注释。该项目的研究部分开发了新的视觉场景理解算法，可以有效地从这种弱注释的视觉数据中学习。主要的新颖之处在于将图像和视频结合在一起。开发的算法可能会在人工智能、安全、农业科学等众多领域产生广泛影响。除了科学影响外，该项目还开展了补充的教育和推广活动。具体来说，它为高中生、本科生和研究生提供指导，教授加州大学戴维斯分校缺乏的新的本科生和研究生计算机视觉课程，并组织了一个关于弱监督视觉场景理解的国际研讨会。该项目开发了新的算法，以两种互补的方式推进弱监督视觉场景理解：(1)与图像和视频共同学习，以利用它们的互补性；(2)从弱监督信号中学习，这些信号超出了标准的语义标签，如时间戳、字幕和相对比较。具体来说，它研究了新的方法来推进任务，如全自动视频对象分割，弱监督对象检测，对象类别的无监督学习，以及挖掘与弱监督信号相关的图像/视频数据中的局部模式。在整个过程中，该项目探索了如何理解和减轻弱标签中的噪声，并克服图像和视频之间的域差异。该奖项反映了美国国家科学基金会的法定使命，并通过使用基金会的知识价值和更广泛的影响审查标准进行评估，被认为值得支持。

项目成果

The Two Dimensions of Worst-case Training and Their Integrated Effect for Out-of-domain Generalization

• DOI: 10.1109/cvpr52688.2022.00941
• 发表时间: 2022-04
• 期刊: 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)
• 作者: Zeyi Huang;Haohan Wang;Dong Huang;Yong Jae Lee;Eric P. Xing

Generating Furry Cars: Disentangling Object Shape and Appearance across Multiple Domains

生成毛茸茸的汽车：跨多个领域理清对象形状和外观

• 发表时间: 2021
• 期刊: International Conference on Learning Representations (ICLR
• 作者: Ojha, Utkarsh;Singh, Krishna Kumar;Lee, Yong Jae
• 通讯作者: Lee, Yong Jae

Collaging Class-specific GANs for Semantic Image Synthesis

• DOI: 10.1109/iccv48922.2021.01415
• 发表时间: 2021-10
• 期刊: 2021 IEEE/CVF International Conference on Computer Vision (ICCV)
• 作者: Yuheng Li;Yijun Li;Jingwan Lu;Eli Shechtman;Yong Jae Lee;Krishna Kumar Singh

Toward Learning Human-aligned Cross-domain Robust Models by Countering Misaligned Features

• 发表时间: 2021-11
• 作者: Haohan Wang;Zeyi Huang;Hanlin Zhang;Eric P. Xing

ELEVATER: A Benchmark and Toolkit for Evaluating Language-Augmented Visual Models

• DOI: 10.48550/arxiv.2204.08790
• 发表时间: 2022-04
• 期刊: ArXiv
• 作者: Chunyuan Li-;Haotian Liu;Liunian Harold Li;Pengchuan Zhang;J. Aneja;Jianwei Yang;Ping Jin;Yong Jae Lee;Houdong Hu;Zicheng Liu;Jianfeng Gao