Collaborative Research: Multimodal Sensing and Analytics at Scale: Algorithms and Applications

在多模态信号和数据中发现高度相关的潜在因素：可扩展的算法和应用在传感，成像和语言processingAbstract：多模态信号和数据自然出现在许多科学和工程领域，我们的数字社会提供了越来越多的机会来收集和提取有用的信息，从这些数据。例如，脑磁共振成像和脑电图是感测大脑活动的两种模式，可以提供同一组患者（实体）的不同“视图”。另一个例子是不同语言中一组给定单词的共现频率。犯罪、贫困、福利、收入、税收、学校、失业和其他类型的社会数据提供了对一组特定城市的不同看法。整合多个视图以提取有意义的公共信息是非常有趣的，并且发现了大量及时的应用-在脑成像，机器翻译，遥感中的景观变化检测和社会科学研究中，仅举几例。 然而，现有的多视图分析工具-特别是（广义）典型相关分析[（G）CCA] -正在努力跟上当今数据集的规模，问题只会变得更糟。此外，复杂的结构和动态性质的一些基本现象没有占在经典的GCCA。该项目将为基于GCCA的多模态传感和分析提供急需的可扩展和灵活的计算工具，从而使各种科学和工程应用受益。它将产生一个框架，允许即插即用的应用程序特定的先验信息，并分布式实现。除了线性和批量GCCA，非线性GCCA和流式GCCA将被考虑。这些对于许多应用来说都是有吸引力和及时的，但是相关的计算工具非常缺乏。在理论和方法方面，GCCA的许多关键方面（例如收敛特性，分布式实现和流式变体）仍然知之甚少。该研究将提供一套高性能的计算工具，这些工具由先进的优化理论和严格的收敛保证支持。该研究将沿着以下协同推进：1）可扩展和随机GCCA算法; 2）分布式，流式和非线性GCCA算法;和3）验证，使用一系列及时和重要的应用在遥感，脑成像，自然语言处理和传感器阵列处理。设计可扩展的，灵活的，流式和非线性GCCA算法对于现代传感和分析问题是非常有意义的，这些问题涉及快速增加的数据量和未知的底层动态。将GCCA用于大规模动态和复杂数据也会带来非常具有挑战性和令人兴奋的建模和优化问题。该奖项反映了NSF的法定使命，并通过使用基金会的知识价值和更广泛的影响审查标准进行评估，被认为值得支持。

Finding highly correlated latent factors in multimodal signals and data: Scalable algorithms and applications in sensing, imaging, and language processingAbstract: Multimodal signals and data arise naturally in many walks of science and engineering, and our digital society presents ever-increasing opportunities to collect and extract useful information from such data. For example, brain magnetic resonance imaging and electro-encephalography are two modes of sensing brain activity that can offer different "views" of the same set of patients (entities). Co-occurrence frequencies of a given set of words in different languages is another example. Crime, poverty, welfare, income, tax, school, unemployment, and other types of social data offer different views of a given set of municipalities. Integrating multiple views to extract meaningful common information is of great interest, and finds a vast amount of timely applications -- in brain imaging, machine translation, landscape change detection in remote sensing, and social science research, to name a few. However, existing multiview analytics tools -- notably (generalized) canonical correlation analysis [(G)CCA] -- are struggling to keep pace with the size of today's datasets, and the problem is only getting worse. Furthermore, the complex structure and dynamic nature of some of the underlying phenomena are not accounted for in classical GCCA. This project will provide much needed scalable and flexible computational tools for GCCA-based multimodal sensing and analytics, thereby benefiting a large variety of scientific and engineering applications. It will produce a framework allowing for plug-and-play incorporation of application-specific prior information, and distributed implementation. Beyond linear and batch GCCA, nonlinear GCCA and streaming GCCA will be considered. These are appealing and timely for many applications, but associated computational tools are sorely missing.In terms of theory and methods, many key aspects of GCCA (such as convergence properties, distributed implementation, and streaming variants) are still poorly understood. The research will provide a set of high-performance computational tools that are backed by advanced optimization theory and rigorous convergence guarantees. The research will evolve along the following synergistic thrusts: 1) scalable and stochastic GCCA algorithms; 2) distributed, streaming and nonlinear GCCA algorithms; and 3) validation, using a series of timely and important applications in remote sensing, brain imaging, natural language processing, and sensor array processing. Devising scalable, flexible, streaming, and nonlinear GCCA algorithms is very well-motivated for modern sensing and analytics problems which involve rapidly increasing amounts of data with unknown underlying dynamics. Using GCCA for large-scale dynamic and complex data also poses very challenging and exciting modeling and optimization problems.This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.