III: Small: Multimodal Machine Learning for Data with Incomplete Modalities

随着数据采集技术的进步，从多个源采集的大量多模态数据被广泛使用，包括文本、图像、视频和音频。这种多模态数据可以提供补充信息，从而揭示应用程序的基本特征。因此，多模态机器学习建立可以处理和关联来自多个模态的信息的模型已经成为一个活跃的研究领域。人们已经开展了广泛的工作来组合联合收割机不同的模态、学习联合表示以利用多个模态的互补性和冗余性以及融合信息以执行预测任务。然而，有效地集成和分析多模态数据仍然是一个具有挑战性的问题，特别是当数据是不完整的。模态缺失是真实世界多模态数据中常见的问题，它可以由各种原因引起，如传感器损坏，数据损坏和记录中的人为错误。这种不完整的数据给多模态机器学习带来了重大挑战。该项目开发了一个多模态机器学习框架，制定了一个新的基本结构，以促进复杂的信息提取和整合，从多模态数据与不完整的模式进行分类和预测。该项目具有推进多模态机器学习理论和实践的巨大潜力，在信息系统、工程和生物医学等目标领域具有重要意义。在技术上，该项目开发了一个框架，该框架利用基于图的结构的强大表示功能来建模异构数据集之间的复杂交互，促进深度信息融合和可持续的数据分析。提出的多模态机器学习方法制定了一个新的基本框架，使复杂的信息提取和集成的多模态数据与不完整的模态。 设计了一种多层次超图结构对不完全多模态数据进行建模，并提出了一种多阶段数据融合框架，通过该框架将具有不同缺失数据条件的所有异构数据点投影到同一嵌入空间中，实现多模态数据的沿着融合.该方法对复杂的模态内和模态间关系进行建模，提取互补的多模态信息，并将多个子空间的信息融合到一个统一的表示中。拟议的研究开发了一种独特的策略，用于学习不完整模态的数据，而无需删除数据或数据插补。当一个或多个模态缺失时，所提出的研究使得来自不完整模态的信息能够有效地包含在学习过程中。此外，所提出的解释方法利用从数据中学习到的丰富语义来解释特定预测决策背后的模型行为。该奖项反映了NSF的法定使命，并通过使用基金会的知识价值和更广泛的影响审查标准进行评估，被认为值得支持。

With the advances in data collection techniques, large amounts of multimodal data collected from multiple sources are widely available, including text, images, video, and audio. Such multimodal data can provide complementary information that can reveal the fundamental characteristics of applications. Thus, multimodal machine learning which builds models that can process and relate information from multiple modalities has become an active research area. Extensive works have been developed to combine different modalities, learn joint representations to exploit complementarity and redundancy of multiple modalities, and fuse information to perform prediction tasks. However, effectively integrating and analyzing multimodal data remains a challenging problem, especially when the data is incomplete. Missing modality is a common issue in real-world multimodal data, which can be caused by various reasons such as sensor damage, data corruption, and human mistakes in recording. Such incomplete data imposes significant challenges to multimodal machine learning. This project develops a multimodal machine learning framework that formulates a new fundamental structure to facilitate the complex information extraction and integration from multimodal data with incomplete modalities for classification and prediction. This project has significant potential to advance the theory and practice of multimodal machine learning, with strong implication in targeted domains such as Information Systems, Engineering, and Biomedicine. Technically, this project develops a framework that exploits the powerful representational features of graph based structure to model the complex interaction between heterogeneous datasets, promoting deep information fusion and sustainable data analysis. The proposed multimodal machine learning approach formulates a new fundamental framework that enables the complex information extraction and integration from multimodal data with incomplete modalities. A multi-level hypergraph structure is designed to model the multimodal data with incompleteness, and a multistage data fusion framework is developed to enable a transductive learning process through which all heterogeneous data points with different missing data conditions are projected into the same embedding space and multi-modalities are fused along the way. The proposed method models the complex intra and inter modality relationships, extract complementary multimodal information, and fuse the information from multiple subspaces to a unified representation. The proposed research develops a unique strategy for learning on data with incomplete modalities, without data deletion or data imputation. The proposed research enables the information from incomplete modalities to be effectively included in the learning process when one or more modalities are missing. Moreover, the proposed interpretation method utilizes the rich semantics learned from the data to interpret the model behaviors behind a particular prediction decision. This increases the transparency of the proposed approach and also contributes towards explainable machine learning.This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.