Collaborative Research: Multimodal Sensing and Analytics at Scale: Algorithms and Applications

在多模信号和数据中寻找高度相关的潜在因素：传感、成像和语言处理中的可扩展算法和应用摘要：多模信号和数据自然地出现在许多科学和工程领域，我们的数字社会提供了越来越多的机会从这些数据中收集和提取有用的信息。例如，脑磁共振成像和脑电是两种感知大脑活动的模式，可以提供同一组患者(实体)的不同“观点”。另一个例子是给定的一组单词在不同语言中的共现频率。犯罪、贫困、福利、收入、税收、学校、失业和其他类型的社会数据提供了对一组给定市政当局的不同看法。集成多个视图以提取有意义的共同信息非常有趣，并发现了大量及时的应用--在脑成像、机器翻译、遥感中的景观变化检测和社会科学研究等方面。然而，现有的多视图分析工具--特别是(广义的)典型相关分析[(G)CCA]--难以跟上当今数据集的规模，而且问题只会变得更糟。此外，一些潜在现象的复杂结构和动态性质在经典的GCCA中没有得到解释。该项目将为基于GCCA的多模式传感和分析提供急需的可扩展和灵活的计算工具，从而使各种科学和工程应用受益。它将产生一个框架，允许即插即用地合并特定于应用程序的先验信息，并进行分布式实施。除了线性GCCA和批GCCA之外，还将考虑非线性GCCA和流GCCA。对于许多应用来说，这些都是吸引人的和及时的，但相关的计算工具严重缺乏。在理论和方法方面，GCCA的许多关键方面(如收敛特性、分布式实现和流变体)仍然知之甚少。这项研究将提供一套以先进的优化理论和严格的收敛保证为后盾的高性能计算工具。这项研究将沿着以下协同方向发展：1)可扩展和随机的GCCA算法；2)分布式、流和非线性的GCCA算法；以及3)验证，使用一系列及时和重要的应用于遥感、脑成像、自然语言处理和传感器阵列处理。设计可扩展的、灵活的、流的和非线性的GCCA算法非常适合于现代传感和分析问题，这些问题涉及到具有未知潜在动态的快速增长的数据量。使用GCCA处理大规模的动态和复杂数据也带来了非常具有挑战性和激动人心的建模和优化问题。该奖项反映了NSF的法定使命，并通过使用基金会的智力优势和更广泛的影响审查标准进行评估，被认为值得支持。

Finding highly correlated latent factors in multimodal signals and data: Scalable algorithms and applications in sensing, imaging, and language processingAbstract: Multimodal signals and data arise naturally in many walks of science and engineering, and our digital society presents ever-increasing opportunities to collect and extract useful information from such data. For example, brain magnetic resonance imaging and electro-encephalography are two modes of sensing brain activity that can offer different "views" of the same set of patients (entities). Co-occurrence frequencies of a given set of words in different languages is another example. Crime, poverty, welfare, income, tax, school, unemployment, and other types of social data offer different views of a given set of municipalities. Integrating multiple views to extract meaningful common information is of great interest, and finds a vast amount of timely applications -- in brain imaging, machine translation, landscape change detection in remote sensing, and social science research, to name a few. However, existing multiview analytics tools -- notably (generalized) canonical correlation analysis [(G)CCA] -- are struggling to keep pace with the size of today's datasets, and the problem is only getting worse. Furthermore, the complex structure and dynamic nature of some of the underlying phenomena are not accounted for in classical GCCA. This project will provide much needed scalable and flexible computational tools for GCCA-based multimodal sensing and analytics, thereby benefiting a large variety of scientific and engineering applications. It will produce a framework allowing for plug-and-play incorporation of application-specific prior information, and distributed implementation. Beyond linear and batch GCCA, nonlinear GCCA and streaming GCCA will be considered. These are appealing and timely for many applications, but associated computational tools are sorely missing.In terms of theory and methods, many key aspects of GCCA (such as convergence properties, distributed implementation, and streaming variants) are still poorly understood. The research will provide a set of high-performance computational tools that are backed by advanced optimization theory and rigorous convergence guarantees. The research will evolve along the following synergistic thrusts: 1) scalable and stochastic GCCA algorithms; 2) distributed, streaming and nonlinear GCCA algorithms; and 3) validation, using a series of timely and important applications in remote sensing, brain imaging, natural language processing, and sensor array processing. Devising scalable, flexible, streaming, and nonlinear GCCA algorithms is very well-motivated for modern sensing and analytics problems which involve rapidly increasing amounts of data with unknown underlying dynamics. Using GCCA for large-scale dynamic and complex data also poses very challenging and exciting modeling and optimization problems.This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.