RI: Small: SM-An Active Approach for Data Engineering to Improve Vision-Language Tasks

RI：小型：SM - 一种改进视觉语言任务的数据工程主动方法

• 批准号: 2132724
• 负责人: Yezhou Yang
• 金额: $ 49.99万
• 依托单位: Arizona State University
• 依托单位国家: 美国
• 项目类别: Continuing Grant
• 财政年份: 2022
• 资助国家: 美国
• 起止时间: 2022-04-01 至 2026-03-31
• 项目状态: 未结题
• 来源: https://www.nsf.gov/awardsearch/showAward?AWD_ID=2132724&HistoricalAwards=false
• 关键词: RI; Small; SM; Active; Approach

Intelligent systems that can robustly process vision and language data are necessary to enable integrated AI applications (such as automated driving, robotic home assistant, etc.) and improve quality of life. However, such systems typically operate in open and highly uncertain environments for which physical and geometric understanding, semantic robustness, and conducting hypothetical reasoning become essential. This project will result in a publicly available software suite that can assist with training and validating robust Vision and Language (V&L) systems. In particular, the resulting semantic transformations will be packaged as an API service that companies and universities could quickly utilize. The resulting benchmark challenges will be made publicly available for further V&L research. Finally, the proposed study will stimulate educational activities at ASU in training graduate and undergraduate students in AI/ML/CV/NLP with a "post-dataset era'" vision. The project will also train 2 Ph.D. students and several master-with-thesis students, develop a new seminar course, recruit underrepresented minority participants at all levels, and reach K-12 students with modules that explain the challenges in developing robust intelligent systems.Robust intelligent systems such as home assistant robots fundamentally depend on highly correlated vision and language systems and fine-grained data alignment. Even though the existing approaches demonstrate success on carefully collected benchmarks, it is not sufficient to establish robustness, reliability, and out-of-distribution generalization for them to be deployed in real-world applications. The project will conduct a systematic study on intelligent and active data engineering to boost their performance and robustness. By investigating a novel and active perspective towards vision and language data engineering, the project will address the following three fundamental research tasks: 1) development of data generators to hallucinate training data from existing ones with low-level vision; 2) with hypothetical actions, and 3) design of training paradigms incorporating the new data generated with the goal of increasing the ultimate systems' generalization capability and robustness.This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.

可以强大的处理视觉和语言数据的智能系统对于启用集成的AI应用程序（例如自动驾驶，机器人家庭助理等）并提高生活质量是必要的。但是，这种系统通常在开放且高度不确定的环境中运行，这些环境的物理和几何理解，语义鲁棒性以及进行假设推理变得至关重要。该项目将导致公开可用的软件套件，可以帮助培训和验证强大的视觉和语言（V＆L）系统。特别是，由此产生的语义转换将被包装为公司和大学可以迅速利用的API服务。由此产生的基准挑战将公开用于进一步的V＆L研究。最后，拟议的研究将在AI/ML/ML/CV/NLP的培训研究生和本科生中刺激ASU的教育活动，其愿景为“后时代”。该项目还将培训2博士学位。学生和几位主持论文的学生开发了新的研讨会课程，在各个级别招募了代表性不足的少数群体参与者，并吸引了具有模块的K-12学生，这些模块可以解释开发强大的智能系统的挑战。RobustIntelligent Systems。诸如家庭助理机器人（例如家庭助理机器人）基本上取决于高度相关的视觉和语言系统和细化的数据和良好的数据和元素。 即使现有的方法在精心收集的基准上表现出成功，但建立鲁棒性，可靠性和分布外的概括不足以在现实世界应用程序中部署。该项目将对智能和主动数据工程进行系统研究，以提高其性能和鲁棒性。 通过调查对视觉和语言数据工程的新颖和积极的观点，该项目将解决以下三个基本研究任务：1）开发数据生成器以幻觉，以幻觉来自具有低级视觉的现有数据的培训数据； 2）采用假设的行动和3）培训范式的设计，结合了生成的新数据，目的是提高最终系统的概括能力和鲁棒性。该奖项反映了NSF的法定任务，并被认为是通过基金会的知识分子优点和更广泛的影响审查标准来通过评估来获得支持的。

项目成果

CRIPP-VQA: Counterfactual Reasoning about Implicit Physical Properties via Video Question Answering

• DOI: 10.48550/arxiv.2211.03779
• 发表时间: 2022-11
• 期刊: ArXiv
• 作者: Maitreya Patel;Tejas Gokhale;Chitta Baral;Yezhou Yang

Semantically Distributed Robust Optimization for Vision-and-Language Inference

• DOI: 10.18653/v1/2022.findings-acl.118
• 发表时间: 2021-10
• 作者: Tejas Gokhale;A. Chaudhary;Pratyay Banerjee;Chitta Baral;Yezhou Yang

Improving Diversity with Adversarially Learned Transformations for Domain Generalization

• DOI: 10.1109/wacv56688.2023.00051
• 发表时间: 2022-06
• 期刊: 2023 IEEE/CVF Winter Conference on Applications of Computer Vision (WACV)
• 作者: Tejas Gokhale;Rushil Anirudh;J. Thiagarajan;B. Kailkhura;Chitta Baral;Yezhou Yang