III: Small: Multimodal Machine Learning for Data with Incomplete Modalities

随着数据收集技术的进步，从多个来源收集的大量多模态数据广泛可用，包括文本、图像、视频和音频。这种多模态数据可以提供能够揭示应用程序基本特征的补充信息。因此，构建能够处理和关联来自多个模态的信息的模型的多模态机器学习已成为一个活跃的研究领域。大量的工作已经发展到结合不同的模式，学习联合表示来利用多模式的互补性和冗余性，并融合信息来执行预测任务。然而，有效地整合和分析多模态数据仍然是一个具有挑战性的问题，特别是在数据不完整的情况下。模态缺失是实际多模态数据中的常见问题，它可能由传感器损坏、数据损坏和记录中的人为错误等各种原因引起。这种不完整的数据给多模态机器学习带来了重大挑战。本项目开发了一个多模态机器学习框架，该框架制定了一个新的基本结构，以促进从具有不完整模态的多模态数据中提取和集成复杂信息进行分类和预测。该项目在推进多模态机器学习的理论和实践方面具有重要的潜力，在信息系统、工程和生物医学等目标领域具有重要的意义。从技术上讲，该项目开发了一个框架，利用基于图的结构的强大表征特征来模拟异构数据集之间的复杂交互，促进深度信息融合和可持续的数据分析。提出的多模态机器学习方法为从不完整模态的多模态数据中提取和集成复杂信息提供了一个新的基本框架。设计了多级超图结构对不完备的多模态数据进行建模，开发了多级数据融合框架，实现了转换学习过程，将具有不同缺失数据条件的异构数据点投影到同一嵌入空间中，并在此过程中融合多模态。该方法对复杂的模态内和模态间关系进行建模，提取互补的多模态信息，并将多个子空间的信息融合为统一的表示。提出的研究开发了一种独特的策略，用于学习具有不完整模式的数据，无需数据删除或数据输入。本文提出的研究能够在一个或多个模态缺失的情况下，将来自不完整模态的信息有效地包含在学习过程中。此外，所提出的解释方法利用从数据中学习到的丰富语义来解释特定预测决策背后的模型行为。这增加了所提出方法的透明度，也有助于实现可解释的机器学习。该奖项反映了美国国家科学基金会的法定使命，并通过使用基金会的知识价值和更广泛的影响审查标准进行评估，被认为值得支持。

With the advances in data collection techniques, large amounts of multimodal data collected from multiple sources are widely available, including text, images, video, and audio. Such multimodal data can provide complementary information that can reveal the fundamental characteristics of applications. Thus, multimodal machine learning which builds models that can process and relate information from multiple modalities has become an active research area. Extensive works have been developed to combine different modalities, learn joint representations to exploit complementarity and redundancy of multiple modalities, and fuse information to perform prediction tasks. However, effectively integrating and analyzing multimodal data remains a challenging problem, especially when the data is incomplete. Missing modality is a common issue in real-world multimodal data, which can be caused by various reasons such as sensor damage, data corruption, and human mistakes in recording. Such incomplete data imposes significant challenges to multimodal machine learning. This project develops a multimodal machine learning framework that formulates a new fundamental structure to facilitate the complex information extraction and integration from multimodal data with incomplete modalities for classification and prediction. This project has significant potential to advance the theory and practice of multimodal machine learning, with strong implication in targeted domains such as Information Systems, Engineering, and Biomedicine. Technically, this project develops a framework that exploits the powerful representational features of graph based structure to model the complex interaction between heterogeneous datasets, promoting deep information fusion and sustainable data analysis. The proposed multimodal machine learning approach formulates a new fundamental framework that enables the complex information extraction and integration from multimodal data with incomplete modalities. A multi-level hypergraph structure is designed to model the multimodal data with incompleteness, and a multistage data fusion framework is developed to enable a transductive learning process through which all heterogeneous data points with different missing data conditions are projected into the same embedding space and multi-modalities are fused along the way. The proposed method models the complex intra and inter modality relationships, extract complementary multimodal information, and fuse the information from multiple subspaces to a unified representation. The proposed research develops a unique strategy for learning on data with incomplete modalities, without data deletion or data imputation. The proposed research enables the information from incomplete modalities to be effectively included in the learning process when one or more modalities are missing. Moreover, the proposed interpretation method utilizes the rich semantics learned from the data to interpret the model behaviors behind a particular prediction decision. This increases the transparency of the proposed approach and also contributes towards explainable machine learning.This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.