CAREER: Automated Multimodal Learning for Healthcare

多通道学习是人工智能(AI)的核心任务之一，其目的是有效地融合和建模多通道数据，以更好地理解我们周围的世界。已经提出了许多多模式融合策略，从手动设计的策略到基于高级自动机器学习(AutoML)的方法。尽管基于AutoML的解决方案优于手工制作的解决方案，但由于它们在模型设计中缺乏泛化能力，并且未能考虑到多模式数据的独特特征，它们仍然远未达到最佳。该项目以多模式医疗预测建模任务为代表，旨在发现和识别通过一种新的学习范式融合多模式数据的最佳方式，即自动化多模式学习，并最大限度地减少人工干预。该项目的成功将在各个领域产生新的基础知识，包括自动化机器学习、多模式深度学习和医疗保健预测建模。新的自动化多模式学习范式将通过自动从数据中搜索新的复杂但最优的融合策略来彻底改变多模式数据挖掘，潜在地激励研究人员和领域专家更好地理解多模式数据。此外，认识到医疗保健领域多模式数据的独特性质带来的独特研究挑战，并提供定制的解决方案，将极大地推动医疗保健预测建模的研究。为了实现这些目标，研究人员建议为自动多模式学习配备能够对多模式健康数据的独特挑战进行建模的能力，包括数据大小变化、噪声和丢失的模式。研究人员还建议为医疗保健信息学及其他领域的不同多模式融合任务验证建议的研究，并收集专家的反馈以完善建议的研究。该项目的结果将提供向自动化多模式数据融合所需的范式转换，影响广泛的研究领域，包括机器学习、数据挖掘和医疗保健信息学。该研究还将为临床实践和其他领域的多模式预测建模做出持久的贡献。生成的数据、源代码和软件工具将提供给世界各地的研究人员。开放平台将加快研究，加强该领域的全球合作，并为学术界、医疗保健组织和健康行业提供长期价值。拟议的教育计划将有助于确保毕业生具备设计和评估机器学习解决方案的能力，并培养K-12学生对计算机科学和信息学的兴趣。这一奖项反映了NSF的法定使命，并通过使用基金会的智力优势和更广泛的影响审查标准进行评估，被认为值得支持。

Multimodal learning is one of the central tasks of artificial intelligence (AI), which aims to effectively fuse and model multimodal data to gain a better understanding of the world around us. Many multimodal fusion strategies have been proposed, ranging from manually designed policies to advanced automated machine learning (AutoML)-based approaches. Although AutoML-based solutions outperform handcrafted ones, they are still far from optimal due to their lack of generalizability in model design and failure to account for the unique characteristics of multimodal data. This project takes the multimodal healthcare predictive modeling task as a representative example, aiming to discover and identify the optimal way to fuse multimodal data via a new learning paradigm, i.e., automated multimodal learning, with minimal human interventions. The success of this project will yield new fundamental knowledge in various fields, including automated machine learning, multimodal deep learning, and healthcare predictive modeling. The new automated multimodal learning paradigm will revolutionize multimodal data mining by automatically searching for new and complex yet optimal fusion strategies from the data, potentially motivating researchers and domain experts to understand the multimodal data better. In addition, recognizing unique research challenges posed by the unique nature of multimodal data in the healthcare domain and providing customized solutions will advance the research of healthcare predictive modeling significantly. To meet these goals, the investigator proposes to equip automated multimodal learning with the ability to model the unique challenges of multimodal health data, including data size variety, noise, and missing modalities. The investigator also proposes to validate the proposed research for different multimodal fusion tasks in healthcare informatics and beyond and gather feedback from experts to refine the proposed research. The results of this project will provide a needed paradigm shift toward automated multimodal data fusion, impacting a broad range of research fields, including machine learning, data mining, and healthcare informatics. The proposed research will also make an enduring contribution to multimodal predictive modeling in clinical practice and other domains. The generated data, source codes, and software tools will be made available to researchers worldwide. The open platform will expedite research, enhance global collaborations in this field, and provide longstanding value for academia, healthcare organizations, and health industries. The proposed education plan will help to ensure that graduates are well equipped to design and evaluate machine learning solutions and cultivate K-12 students' interest in computer science and informatics. It will also lead to a more diverse population of undergraduate research assistants and enhance collaboration and networking among graduate students.This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.