CIF: AF: Small: Foundations of Multimodal Information Integration

CIF：AF：小型：多模式信息集成的基础

• 批准号: 1712867
• 负责人: Guillermo Sapiro
• 金额: $ 43.17万
• 依托单位: Duke University
• 依托单位国家: 美国
• 项目类别: Standard Grant
• 财政年份: 2017
• 资助国家: 美国
• 起止时间: 2017-09-01 至 2023-09-30
• 项目状态: 已结题
• 来源: https://www.nsf.gov/awardsearch/showAward?AWD_ID=1712867&HistoricalAwards=false
• 关键词: CIF; AF; Small; Foundations; Multimodal

Data comes in all forms, including visual, text, medical records, and comments on social medial. This diverse (multimodal) data is critical when data is scarce, noisy, and uncertain. Different modalities can improve joint inference and decision making and allow producing (at extremely low-cost) results that were possible only with high-end devices and techniques before. In addition, inferring a condition from unexpected data sources is of paramount importance in disciplines ranging from marketing to health-care and defense. This project addresses these fundamental challenges with new mathematical and computational tools. Through new collaborations, the project has access to unique data and problems of significant impact in human well-being. A related online class also continues to grow and develop, with over 120,000 students so far. A unique summer immersion program will also involve undergraduate students in multimodal data science research. The exploitation of multimodal data is one of the unifying themes of this project. A further unifying theme is the underlying mathematical foundation: subspace modeling and embedding. Tools from subspace modeling in the form of learning multimodal low-rank representations, modeling multimodal sparse networks, and solving for big data matrix decompositions are here developed. A third unifying motif of this work is the ubiquitous consideration of computational efficiency. All the above is developed in three major components: classification and recognition, data augmentation, and network analysis. The project addresses critical problems such as multimodal face recognition, dynamic multimodal graph inference, gaze analysis, multimodal network analysis, and non-negative matrix factorization. The overall goal is to efficiently exploit and integrate multimodal data to help in joint inference and decision making.

数据以各种形式出现，包括视觉、文本、医疗记录和社交媒体上的评论。当数据稀缺、噪声和不确定时，这种多样化(多模式)数据至关重要。不同的模式可以改进联合推理和决策，并允许(以极低的成本)产生以前只有高端设备和技术才能实现的结果。此外，从意想不到的数据来源推断疾病在从营销到医疗保健和国防等领域都是至关重要的。这个项目用新的数学和计算工具解决了这些根本挑战。通过新的合作，该项目获得了独特的数据和对人类福祉有重大影响的问题。一个相关的在线课程也在继续增长和发展，到目前为止已经有超过12万名学生。一个独特的暑期沉浸项目还将让本科生参与多模式数据科学研究。多模式数据的开发是该项目的统一主题之一。另一个统一的主题是基本的数学基础：子空间建模和嵌入。这里开发了从子空间建模的工具，这些工具包括学习多模低阶表示、建模多模稀疏网络以及求解大数据矩阵分解。这项工作的第三个统一主题是对计算效率的普遍考虑。所有这些都由三个主要部分组成：分类和识别、数据增强和网络分析。该项目解决了多模式人脸识别、动态多模式图形推理、凝视分析、多模式网络分析和非负矩阵分解等关键问题。总体目标是有效地利用和集成多模式数据，以帮助联合推理和决策。

项目成果

Measuring robustness of brain networks in autism spectrum disorder with Ricci curvature

用里奇曲率测量自闭症谱系障碍大脑网络的鲁棒性

• DOI: 10.1101/722025
• 发表时间: 2020
• 期刊: Scientific report
• 作者: A. Simhal, K. Carpenter

A Scalable Off-the-Shelf Framework for Measuring Patterns of Attention in Young Children and its Application in Autism Spectrum Disorder.

• DOI: 10.1109/taffc.2018.2890610
• 发表时间: 2021-07
• 期刊: IEEE TRANSACTIONS ON AFFECTIVE COMPUTING
• 影响因子: 11.2
• 作者: Bovery, Matthieu;Dawson, Geraldine;Hashemi, Jordan;Sapiro, Guillermo
• 通讯作者: Sapiro, Guillermo

Using text to teach image retrieval

使用文本教授图像检索

• DOI: 10.1109/cvprw53098.2021.00180
• 发表时间: 2021
• 期刊: CVPR 2021 Workshop
• 作者: H. Dong, Z. Wang

Atypical postural control can be detected via computer vision analysis in toddlers with autism spectrum disorder.

• DOI: 10.1038/s41598-018-35215-8
• 发表时间: 2018-11-19
• 期刊: Scientific reports
• 影响因子: 4.6
• 作者: Dawson G;Campbell K;Hashemi J;Lippmann SJ;Smith V;Carpenter K;Egger H;Espinosa S;Vermeer S;Baker J;Sapiro G
• 通讯作者: Sapiro G

A Computational Synaptic Antibody Characterization Tool for Array Tomography.

• DOI: 10.3389/fnana.2018.00051
• 发表时间: 2018
• 期刊: Frontiers in neuroanatomy
• 影响因子: 2.9
• 作者: Simhal AK;Gong B;Trimmer JS;Weinberg RJ;Smith SJ;Sapiro G;Micheva KD
• 通讯作者: Micheva KD