RI: Small: SM-An Active Approach for Data Engineering to Improve Vision-Language Tasks

RI：小型：SM - 一种改进视觉语言任务的数据工程主动方法

• 批准号: 2132724
• 负责人: Yezhou Yang
• 金额: $ 49.99万
• 依托单位: Arizona State University
• 依托单位国家: 美国
• 项目类别: Continuing Grant
• 财政年份: 2022
• 资助国家: 美国
• 起止时间: 2022-04-01 至 2026-03-31
• 项目状态: 未结题
• 来源: https://www.nsf.gov/awardsearch/showAward?AWD_ID=2132724&HistoricalAwards=false
• 关键词: RI; Small; SM; Active; Approach

Intelligent systems that can robustly process vision and language data are necessary to enable integrated AI applications (such as automated driving, robotic home assistant, etc.) and improve quality of life. However, such systems typically operate in open and highly uncertain environments for which physical and geometric understanding, semantic robustness, and conducting hypothetical reasoning become essential. This project will result in a publicly available software suite that can assist with training and validating robust Vision and Language (V&L) systems. In particular, the resulting semantic transformations will be packaged as an API service that companies and universities could quickly utilize. The resulting benchmark challenges will be made publicly available for further V&L research. Finally, the proposed study will stimulate educational activities at ASU in training graduate and undergraduate students in AI/ML/CV/NLP with a "post-dataset era'" vision. The project will also train 2 Ph.D. students and several master-with-thesis students, develop a new seminar course, recruit underrepresented minority participants at all levels, and reach K-12 students with modules that explain the challenges in developing robust intelligent systems.Robust intelligent systems such as home assistant robots fundamentally depend on highly correlated vision and language systems and fine-grained data alignment. Even though the existing approaches demonstrate success on carefully collected benchmarks, it is not sufficient to establish robustness, reliability, and out-of-distribution generalization for them to be deployed in real-world applications. The project will conduct a systematic study on intelligent and active data engineering to boost their performance and robustness. By investigating a novel and active perspective towards vision and language data engineering, the project will address the following three fundamental research tasks: 1) development of data generators to hallucinate training data from existing ones with low-level vision; 2) with hypothetical actions, and 3) design of training paradigms incorporating the new data generated with the goal of increasing the ultimate systems' generalization capability and robustness.This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.

能够稳健处理视觉和语言数据的智能系统是实现集成AI应用（如自动驾驶、机器人家庭助理等）的必要条件。提高生活质量。然而，这样的系统通常在开放和高度不确定的环境中运行，对于这些环境，物理和几何理解、语义鲁棒性和进行假设推理变得至关重要。该项目将产生一个公开的软件套件，可以帮助培训和验证强大的视觉和语言（V L）系统。特别是，最终的语义转换将被打包为公司和大学可以快速使用的API服务。由此产生的基准挑战将公开提供进一步的V L研究。最后，拟议的研究将刺激亚利桑那州立大学的教育活动，以“后数据集时代”的眼光培训AI/ML/CV/NLP的研究生和本科生。该项目还将培养2名博士学生和几个硕士论文的学生，开发一个新的研讨会课程，招募代表性不足的少数民族参与者在各个级别，并达到K-12学生与模块，解释在开发强大的智能系统的挑战.强大的智能系统，如家庭助理机器人从根本上依赖于高度相关的视觉和语言系统和细粒度的数据对齐. 尽管现有的方法在仔细收集的基准测试中取得了成功，但还不足以建立鲁棒性，可靠性和分布泛化，以便在现实世界的应用程序中部署它们。该项目将对智能和主动数据工程进行系统研究，以提高其性能和鲁棒性。 通过研究视觉和语言数据工程的新颖和积极的观点，该项目将解决以下三个基本研究任务：1）开发数据生成器，以从现有的低水平视觉数据中产生幻觉训练数据;（2）假设行为，和3）训练范例的设计，结合新的数据产生的目标，提高最终系统的泛化能力和鲁棒性。这该奖项反映了NSF的法定使命，并通过使用基金会的知识价值和更广泛的影响审查标准进行评估，被认为值得支持。

项目成果

CRIPP-VQA: Counterfactual Reasoning about Implicit Physical Properties via Video Question Answering

• DOI: 10.48550/arxiv.2211.03779
• 发表时间: 2022-11
• 期刊: ArXiv
• 作者: Maitreya Patel;Tejas Gokhale;Chitta Baral;Yezhou Yang

Semantically Distributed Robust Optimization for Vision-and-Language Inference

• DOI: 10.18653/v1/2022.findings-acl.118
• 发表时间: 2021-10
• 作者: Tejas Gokhale;A. Chaudhary;Pratyay Banerjee;Chitta Baral;Yezhou Yang

Improving Diversity with Adversarially Learned Transformations for Domain Generalization

• DOI: 10.1109/wacv56688.2023.00051
• 发表时间: 2022-06
• 期刊: 2023 IEEE/CVF Winter Conference on Applications of Computer Vision (WACV)
• 作者: Tejas Gokhale;Rushil Anirudh;J. Thiagarajan;B. Kailkhura;Chitta Baral;Yezhou Yang