III: Small: Multimodal Machine Learning for Data with Incomplete Modalities

随着数据收集技术的进步，从多个来源收集的大量多模式数据广泛可用，包括文本、图像、视频和音频。这种多模式数据可以提供补充信息，可以揭示应用程序的基本特征。因此，建立能够处理和关联来自多个通道的信息的模型的多通道机器学习已经成为一个活跃的研究领域。已经开展了广泛的工作来组合不同的模态，学习联合表示以利用多个模态的互补性和冗余性，并融合信息来执行预测任务。然而，有效地集成和分析多模式数据仍然是一个具有挑战性的问题，特别是在数据不完整的情况下。丢失通道是现实世界多通道数据中常见的问题，它可能由传感器损坏、数据损坏和记录过程中的人为错误等各种原因造成。这种不完整的数据给多模式机器学习带来了巨大的挑战。这个项目开发了一个多模式机器学习框架，它制定了一个新的基本结构，以促进从分类和预测的不完全模式的多模式数据中提取和集成复杂的信息。该项目对推进多模式机器学习的理论和实践具有重要的潜力，在信息系统、工程和生物医学等目标领域具有很强的应用价值。在技术上，该项目开发了一个框架，该框架利用基于图的结构的强大表示特性来建模不同数据集之间的复杂交互，促进深度信息融合和可持续数据分析。提出的多模式机器学习方法形成了一个新的基本框架，能够从具有不完全模式的多模式数据中提取和集成复杂的信息。设计了一种多层超图结构来对不完全的多通道数据进行建模，并开发了一个多阶段数据融合框架来实现一个转导学习过程，通过该过程，所有具有不同缺失数据条件的异质数据点被投影到同一嵌入空间中，并在此过程中对多通道进行融合。该方法对复杂的通道内和通道间关系进行建模，提取互补的多通道信息，并将多个子空间的信息融合到统一的表示中。这项拟议的研究开发了一种独特的策略，用于学习具有不完整模式的数据，而不删除或归因于数据。所提出的研究使得当一个或多个通道丢失时，来自不完整通道的信息能够有效地包括在学习过程中。此外，所提出的解释方法利用从数据中学习的丰富语义来解释特定预测决策背后的模型行为。这增加了建议方法的透明度，也有助于实现可解释的机器学习。该奖项反映了NSF的法定使命，并通过使用基金会的智力优势和更广泛的影响审查标准进行评估，被认为值得支持。

With the advances in data collection techniques, large amounts of multimodal data collected from multiple sources are widely available, including text, images, video, and audio. Such multimodal data can provide complementary information that can reveal the fundamental characteristics of applications. Thus, multimodal machine learning which builds models that can process and relate information from multiple modalities has become an active research area. Extensive works have been developed to combine different modalities, learn joint representations to exploit complementarity and redundancy of multiple modalities, and fuse information to perform prediction tasks. However, effectively integrating and analyzing multimodal data remains a challenging problem, especially when the data is incomplete. Missing modality is a common issue in real-world multimodal data, which can be caused by various reasons such as sensor damage, data corruption, and human mistakes in recording. Such incomplete data imposes significant challenges to multimodal machine learning. This project develops a multimodal machine learning framework that formulates a new fundamental structure to facilitate the complex information extraction and integration from multimodal data with incomplete modalities for classification and prediction. This project has significant potential to advance the theory and practice of multimodal machine learning, with strong implication in targeted domains such as Information Systems, Engineering, and Biomedicine. Technically, this project develops a framework that exploits the powerful representational features of graph based structure to model the complex interaction between heterogeneous datasets, promoting deep information fusion and sustainable data analysis. The proposed multimodal machine learning approach formulates a new fundamental framework that enables the complex information extraction and integration from multimodal data with incomplete modalities. A multi-level hypergraph structure is designed to model the multimodal data with incompleteness, and a multistage data fusion framework is developed to enable a transductive learning process through which all heterogeneous data points with different missing data conditions are projected into the same embedding space and multi-modalities are fused along the way. The proposed method models the complex intra and inter modality relationships, extract complementary multimodal information, and fuse the information from multiple subspaces to a unified representation. The proposed research develops a unique strategy for learning on data with incomplete modalities, without data deletion or data imputation. The proposed research enables the information from incomplete modalities to be effectively included in the learning process when one or more modalities are missing. Moreover, the proposed interpretation method utilizes the rich semantics learned from the data to interpret the model behaviors behind a particular prediction decision. This increases the transparency of the proposed approach and also contributes towards explainable machine learning.This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.