CAREER: Automated Multimodal Learning for Healthcare

多模态学习是人工智能（AI）的核心任务之一，其目的是有效地融合和建模多模态数据，以更好地了解我们周围的世界。已经提出了许多多模态融合策略，从手动设计的策略到基于高级自动化机器学习（AutoML）的方法。尽管基于automl的解决方案优于手工制作的解决方案，但由于它们在模型设计中缺乏通用性，并且未能考虑到多模态数据的独特特征，它们仍远未达到最佳效果。本项目以多模态医疗预测建模任务为代表，旨在通过一种新的学习范式（即自动化多模态学习）发现和确定融合多模态数据的最佳方法，同时减少人为干预。该项目的成功将在各个领域产生新的基础知识，包括自动化机器学习、多模态深度学习和医疗保健预测建模。新的自动化多模态学习范式将通过自动从数据中搜索新的、复杂的、最优的融合策略，彻底改变多模态数据挖掘，潜在地激励研究人员和领域专家更好地理解多模态数据。此外，认识到医疗保健领域中多模态数据的独特性质所带来的独特研究挑战，并提供定制的解决方案，将显著推进医疗保健预测建模的研究。为了实现这些目标，研究者建议为自动化多模式学习提供建模多模式健康数据的独特挑战的能力，包括数据大小变化、噪声和缺失模式。研究者还建议在医疗信息学和其他领域的不同多模态融合任务中验证所提议的研究，并收集专家的反馈以完善所提议的研究。该项目的结果将为自动化多模态数据融合提供所需的范式转变，影响广泛的研究领域，包括机器学习、数据挖掘和医疗保健信息学。提出的研究也将为临床实践和其他领域的多模态预测建模做出持久的贡献。生成的数据、源代码和软件工具将提供给全世界的研究人员。该开放平台将加速研究，加强该领域的全球合作，并为学术界、医疗保健组织和卫生产业提供长期价值。拟议的教育计划将有助于确保毕业生有能力设计和评估机器学习解决方案，并培养K-12学生对计算机科学和信息学的兴趣。它还将导致本科生研究助理的人口更加多样化，并加强研究生之间的合作和网络。该奖项反映了美国国家科学基金会的法定使命，并通过使用基金会的知识价值和更广泛的影响审查标准进行评估，被认为值得支持。

Multimodal learning is one of the central tasks of artificial intelligence (AI), which aims to effectively fuse and model multimodal data to gain a better understanding of the world around us. Many multimodal fusion strategies have been proposed, ranging from manually designed policies to advanced automated machine learning (AutoML)-based approaches. Although AutoML-based solutions outperform handcrafted ones, they are still far from optimal due to their lack of generalizability in model design and failure to account for the unique characteristics of multimodal data. This project takes the multimodal healthcare predictive modeling task as a representative example, aiming to discover and identify the optimal way to fuse multimodal data via a new learning paradigm, i.e., automated multimodal learning, with minimal human interventions. The success of this project will yield new fundamental knowledge in various fields, including automated machine learning, multimodal deep learning, and healthcare predictive modeling. The new automated multimodal learning paradigm will revolutionize multimodal data mining by automatically searching for new and complex yet optimal fusion strategies from the data, potentially motivating researchers and domain experts to understand the multimodal data better. In addition, recognizing unique research challenges posed by the unique nature of multimodal data in the healthcare domain and providing customized solutions will advance the research of healthcare predictive modeling significantly. To meet these goals, the investigator proposes to equip automated multimodal learning with the ability to model the unique challenges of multimodal health data, including data size variety, noise, and missing modalities. The investigator also proposes to validate the proposed research for different multimodal fusion tasks in healthcare informatics and beyond and gather feedback from experts to refine the proposed research. The results of this project will provide a needed paradigm shift toward automated multimodal data fusion, impacting a broad range of research fields, including machine learning, data mining, and healthcare informatics. The proposed research will also make an enduring contribution to multimodal predictive modeling in clinical practice and other domains. The generated data, source codes, and software tools will be made available to researchers worldwide. The open platform will expedite research, enhance global collaborations in this field, and provide longstanding value for academia, healthcare organizations, and health industries. The proposed education plan will help to ensure that graduates are well equipped to design and evaluate machine learning solutions and cultivate K-12 students' interest in computer science and informatics. It will also lead to a more diverse population of undergraduate research assistants and enhance collaboration and networking among graduate students.This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.