CAREER: Structure-Preserving Multimodal Alignment between Vision and Language

人工智能（AI）的一个重大挑战是能够处理多模态视觉和语言数据，同时保持这些模态之间的关系，以便维持不同模态之间的联系。当前的机器学习系统不能完全掌握人类视觉和语言中的结构和关系，因此在可解释性、效率、可测量性和因果关系方面难以产生预期的结果。该项目解决了机器学习中的基本多模态对齐问题，并将推进计算机视觉和自然语言处理的研究，特别是在多模态视觉语言生成和理解的颠覆性创新领域。它将导致视觉和语言的理论认识和实际应用的突破。在这个项目下开发的技术同样可以用于跨模态连接不同类型的潜在结构，而不仅仅局限于视觉和语言。这对于负责任的人工智能在科学领域的应用将是非常有益的，因为人们不仅想了解数据中的关系，还想了解数据的结构和因果解释。这种理解对于减少机器学习模型所表现出的人口偏见也至关重要。通过教育、开源和推广活动，该项目将对从K-12到研究生的各级学生进行人工智能培训和教育，推进理论视觉和语言课程，减少偏见，并进一步实现人工智能的民主化。保持结构是理解如何使机器学习模型更好、更可靠的重要组成部分。本项目旨在通过保留结构的潜在空间对齐，在多模态视觉和语言建模方面取得新的重大科学进展，以建立视觉和语言之间的桥梁。该项目旨在增加语言和视觉嵌入的结构保护性质，并在两种潜在表征之间开发一种保留底层结构的地图。特别是，该项目将通过以下四个重点来实现这些目标：(I)开发保留结构的潜在表征和视觉与语言之间的映射；（二）通过潜在结构提高学习和数据效率；（三）通过结构信息制定新的评价指标，以提高可测量性；（四）建立因果表示和解释框架。该奖项反映了美国国家科学基金会的法定使命，并通过使用基金会的知识价值和更广泛的影响审查标准进行评估，被认为值得支持。

A grand challenge in artificial intelligence (AI) is to be able to process multimodal vision and language data, while preserving relationships across such modalities so that the linkages between the different modalities is sustained. Current machine learning systems do not fully grasp the structures and relationships that exist within human vision and language, and thus have difficulties producing the desired outcomes in terms of interpretability, efficiency, measurability, and causality. This project tackles the fundamental multimodal alignment problem in machine learning and will advance research in both computer vision and natural language processing, especially in the disruptive innovation areas of multimodal vision-language generation and understanding. It will lead to breakthroughs in both theoretical understanding as well as practical applications of vision and language. The techniques developed under this project could similarly be used to connect different types of latent structures across modalities and are not limited to vision and language. This would be extremely beneficial for responsible AI applications in the sciences, where people not only want to understand the relationship in data, but the structure and causal explanations. Such an understanding is also critical for reducing demographic biases that machine learning models exhibit. Through education, open-sourcing and outreach activities, this project will train and educate students of all levels - from K-12 to graduate - in AI, advance theoretical vision and language courses, reduce bias, and further democratize AI.Preserving structure is an essential component of understanding how to make machine learning models better and more reliable. This project aims to create novel and significant scientific advances in multimodal vision and language modeling with structure-preserving latent space alignment to build a bridge between vision and language. The project aims to increase the structural preserving nature for linguistic and visual embeddings and develop a map between the two latent representations that preserves the underlying structures. In particular, the project will achieve these goals through four thrusts: (I) Developing structure-preserving latent representations and mapping between vision and language; (II) Improving learning and data efficiency through latent structures; (III) Develop novel evaluation metrics through structural information to improve measurability; (IV) Develop a causal representation and interpretation framework.This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.