CIF: AF: Small: Foundations of Multimodal Information Integration

数据有各种形式，包括视觉、文本、医疗记录和社交媒体上的评论。当数据稀缺、嘈杂和不确定时，这种多样化（多模态）数据至关重要。不同的模式可以改善联合推理和决策，并允许（以极低的成本）产生以前只有高端设备和技术才能实现的结果。此外，从意想不到的数据来源推断病情在从市场营销到保健和国防等学科中至关重要。这个项目用新的数学和计算工具来解决这些基本的挑战。通过新的合作，该项目获得了对人类福祉有重大影响的独特数据和问题。相关的在线课程也在不断发展壮大，目前已有超过12万名学生。一个独特的夏季浸入式项目也将涉及本科生的多模式数据科学研究。利用多模态数据是这个项目的统一主题之一。一个进一步的统一主题是潜在的数学基础：子空间建模和嵌入。这里开发了以学习多模态低秩表示、建模多模态稀疏网络和求解大数据矩阵分解为形式的子空间建模工具。这项工作的第三个统一主题是对计算效率的普遍考虑。所有这些都分为三个主要部分：分类与识别、数据增强和网络分析。该项目解决关键问题，如多模态人脸识别，动态多模态图推理，凝视分析，多模态网络分析和非负矩阵分解。总体目标是有效地利用和集成多模态数据，以帮助联合推理和决策。

Data comes in all forms, including visual, text, medical records, and comments on social medial. This diverse (multimodal) data is critical when data is scarce, noisy, and uncertain. Different modalities can improve joint inference and decision making and allow producing (at extremely low-cost) results that were possible only with high-end devices and techniques before. In addition, inferring a condition from unexpected data sources is of paramount importance in disciplines ranging from marketing to health-care and defense. This project addresses these fundamental challenges with new mathematical and computational tools. Through new collaborations, the project has access to unique data and problems of significant impact in human well-being. A related online class also continues to grow and develop, with over 120,000 students so far. A unique summer immersion program will also involve undergraduate students in multimodal data science research. The exploitation of multimodal data is one of the unifying themes of this project. A further unifying theme is the underlying mathematical foundation: subspace modeling and embedding. Tools from subspace modeling in the form of learning multimodal low-rank representations, modeling multimodal sparse networks, and solving for big data matrix decompositions are here developed. A third unifying motif of this work is the ubiquitous consideration of computational efficiency. All the above is developed in three major components: classification and recognition, data augmentation, and network analysis. The project addresses critical problems such as multimodal face recognition, dynamic multimodal graph inference, gaze analysis, multimodal network analysis, and non-negative matrix factorization. The overall goal is to efficiently exploit and integrate multimodal data to help in joint inference and decision making.