RI: Small: SM-An Active Approach for Data Engineering to Improve Vision-Language Tasks

能够稳健地处理视觉和语言数据的智能系统是实现集成人工智能应用（如自动驾驶、机器人家庭助理等）和提高生活质量所必需的。然而，这样的系统通常在开放和高度不确定的环境中运行，因此物理和几何理解、语义鲁棒性和进行假设推理是必不可少的。该项目将产生一个公开可用的软件套件，可以帮助培训和验证健壮的视觉和语言（V&amp；L）系统。特别是，结果的语义转换将打包为公司和大学可以快速利用的API服务。由此产生的基准挑战将公开提供给进一步的V&amp；L研究。最后，拟议的研究将刺激亚利桑那州立大学的教育活动，以“后数据集时代”的愿景培养AI/ML/CV/NLP的研究生和本科生。该项目还将培养2名博士生和几名有论文的硕士生，开发一门新的研讨会课程，在各个层面招募未被充分代表的少数民族参与者，并向K-12学生提供解释开发强大智能系统所面临挑战的模块。强大的智能系统，如家庭助理机器人，从根本上依赖于高度相关的视觉和语言系统以及细粒度的数据对齐。尽管现有的方法在仔细收集的基准测试中证明了成功，但要在实际应用程序中部署它们，建立健壮性、可靠性和分布外泛化是不够的。该项目将对智能和主动数据工程进行系统研究，以提高其性能和鲁棒性。通过研究视觉和语言数据工程的一个新颖和积极的视角，该项目将解决以下三个基本研究任务：1)开发数据生成器，从现有的低水平视觉中产生幻觉训练数据；2)假设行为；3)设计包含新生成数据的训练范式，以提高最终系统的泛化能力和鲁棒性。该奖项反映了美国国家科学基金会的法定使命，并通过使用基金会的知识价值和更广泛的影响审查标准进行评估，被认为值得支持。

Intelligent systems that can robustly process vision and language data are necessary to enable integrated AI applications (such as automated driving, robotic home assistant, etc.) and improve quality of life. However, such systems typically operate in open and highly uncertain environments for which physical and geometric understanding, semantic robustness, and conducting hypothetical reasoning become essential. This project will result in a publicly available software suite that can assist with training and validating robust Vision and Language (V&L) systems. In particular, the resulting semantic transformations will be packaged as an API service that companies and universities could quickly utilize. The resulting benchmark challenges will be made publicly available for further V&L research. Finally, the proposed study will stimulate educational activities at ASU in training graduate and undergraduate students in AI/ML/CV/NLP with a "post-dataset era'" vision. The project will also train 2 Ph.D. students and several master-with-thesis students, develop a new seminar course, recruit underrepresented minority participants at all levels, and reach K-12 students with modules that explain the challenges in developing robust intelligent systems.Robust intelligent systems such as home assistant robots fundamentally depend on highly correlated vision and language systems and fine-grained data alignment. Even though the existing approaches demonstrate success on carefully collected benchmarks, it is not sufficient to establish robustness, reliability, and out-of-distribution generalization for them to be deployed in real-world applications. The project will conduct a systematic study on intelligent and active data engineering to boost their performance and robustness. By investigating a novel and active perspective towards vision and language data engineering, the project will address the following three fundamental research tasks: 1) development of data generators to hallucinate training data from existing ones with low-level vision; 2) with hypothetical actions, and 3) design of training paradigms incorporating the new data generated with the goal of increasing the ultimate systems' generalization capability and robustness.This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.