RI: Small: SM-An Active Approach for Data Engineering to Improve Vision-Language Tasks

RI：小型：SM - 一种改进视觉语言任务的数据工程主动方法

• 批准号: 2132724
• 负责人: Yezhou Yang
• 金额: $ 49.99万
• 依托单位: Arizona State University
• 依托单位国家: 美国
• 项目类别: Continuing Grant
• 财政年份: 2022
• 资助国家: 美国
• 起止时间: 2022-04-01 至 2026-03-31
• 项目状态: 未结题
• 来源: https://www.nsf.gov/awardsearch/showAward?AWD_ID=2132724&HistoricalAwards=false
• 关键词: RI; Small; SM; Active; Approach

Intelligent systems that can robustly process vision and language data are necessary to enable integrated AI applications (such as automated driving, robotic home assistant, etc.) and improve quality of life. However, such systems typically operate in open and highly uncertain environments for which physical and geometric understanding, semantic robustness, and conducting hypothetical reasoning become essential. This project will result in a publicly available software suite that can assist with training and validating robust Vision and Language (V&L) systems. In particular, the resulting semantic transformations will be packaged as an API service that companies and universities could quickly utilize. The resulting benchmark challenges will be made publicly available for further V&L research. Finally, the proposed study will stimulate educational activities at ASU in training graduate and undergraduate students in AI/ML/CV/NLP with a "post-dataset era'" vision. The project will also train 2 Ph.D. students and several master-with-thesis students, develop a new seminar course, recruit underrepresented minority participants at all levels, and reach K-12 students with modules that explain the challenges in developing robust intelligent systems.Robust intelligent systems such as home assistant robots fundamentally depend on highly correlated vision and language systems and fine-grained data alignment. Even though the existing approaches demonstrate success on carefully collected benchmarks, it is not sufficient to establish robustness, reliability, and out-of-distribution generalization for them to be deployed in real-world applications. The project will conduct a systematic study on intelligent and active data engineering to boost their performance and robustness. By investigating a novel and active perspective towards vision and language data engineering, the project will address the following three fundamental research tasks: 1) development of data generators to hallucinate training data from existing ones with low-level vision; 2) with hypothetical actions, and 3) design of training paradigms incorporating the new data generated with the goal of increasing the ultimate systems' generalization capability and robustness.This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.

能够稳健地处理视觉和语言数据的智能系统对于实现集成的人工智能应用程序(如自动驾驶、机器人家务助理等)是必要的。并提高生活质量。然而，这样的系统通常在开放和高度不确定的环境中运行，对这些环境来说，物理和几何理解、语义健壮性和进行假设推理变得必不可少。该项目将形成一个公开可用的软件套件，帮助培训和验证Robust Vision and Language(V&Amp；L)系统。特别是，产生的语义转换将被打包为公司和大学可以快速利用的API服务。由此产生的基准挑战将公之于众，以供V&A；L进一步研究。最后，拟议的研究将促进亚利桑那州立大学的教育活动，培训研究生和本科生在AI/ML/CV/NLP方面具有“后数据集时代”的愿景。该项目还将培训两名博士生和几名硕士研究生，开发一门新的研讨会课程，招募各级代表性不足的少数族裔参与者，并向K-12学生提供解释开发强大智能系统的挑战的模块。家庭助理机器人等强健智能系统基本上依赖于高度关联的视觉和语言系统以及细粒度的数据对齐。尽管现有方法在仔细收集的基准上显示了成功，但还不足以建立健壮性、可靠性和分布外的泛化，以便将它们部署在现实世界的应用程序中。该项目将对智能和主动数据工程进行系统研究，以提高其性能和健壮性。通过研究视觉和语言数据工程的新的和积极的视角，该项目将解决以下三项基本研究任务：1)开发数据生成器，以幻觉来自低水平视觉的现有训练数据；2)使用假设行动；以及3)设计训练范例，以增加最终系统的泛化能力和健壮性。该奖项反映了NSF的法定使命，并通过使用基金会的智力优势和更广泛的影响审查标准进行评估，被认为值得支持。

项目成果

CRIPP-VQA: Counterfactual Reasoning about Implicit Physical Properties via Video Question Answering

• DOI: 10.48550/arxiv.2211.03779
• 发表时间: 2022-11
• 期刊: ArXiv
• 作者: Maitreya Patel;Tejas Gokhale;Chitta Baral;Yezhou Yang

Semantically Distributed Robust Optimization for Vision-and-Language Inference

• DOI: 10.18653/v1/2022.findings-acl.118
• 发表时间: 2021-10
• 作者: Tejas Gokhale;A. Chaudhary;Pratyay Banerjee;Chitta Baral;Yezhou Yang

Improving Diversity with Adversarially Learned Transformations for Domain Generalization

• DOI: 10.1109/wacv56688.2023.00051
• 发表时间: 2022-06
• 期刊: 2023 IEEE/CVF Winter Conference on Applications of Computer Vision (WACV)
• 作者: Tejas Gokhale;Rushil Anirudh;J. Thiagarajan;B. Kailkhura;Chitta Baral;Yezhou Yang