Collaborative Research: Multimodal Sensing and Analytics at Scale: Algorithms and Applications

摘要：多模态信号和数据自然出现在科学和工程的许多领域，我们的数字社会提供了越来越多的机会从这些数据中收集和提取有用的信息。例如，脑磁共振成像和脑电图是感知大脑活动的两种模式，可以为同一组患者（实体）提供不同的“视图”。不同语言中给定的一组单词的共现频率是另一个例子。犯罪、贫困、福利、收入、税收、学校、失业和其他类型的社会数据提供了一组给定市政当局的不同观点。整合多个视图以提取有意义的共同信息是非常有趣的，并且在脑成像，机器翻译，遥感景观变化检测和社会科学研究中找到了大量及时的应用，仅举几例。然而，现有的多视图分析工具——尤其是（广义）典型相关分析[(G)CCA]——正在努力跟上当今数据集的规模，而且问题只会变得更糟。此外，一些潜在现象的复杂结构和动态性质在经典GCCA中没有得到考虑。该项目将为基于gca的多模态传感和分析提供急需的可扩展和灵活的计算工具，从而使各种科学和工程应用受益。它将产生一个框架，允许将特定于应用程序的先验信息即插即用合并，并进行分布式实现。除了线性和批量GCCA之外，还将考虑非线性GCCA和流GCCA。对于许多应用程序来说，这些都很有吸引力，也很及时，但相关的计算工具却严重缺失。就理论和方法而言，GCCA的许多关键方面（如收敛特性、分布式实现和流变体）仍然知之甚少。该研究将提供一套高性能的计算工具，以先进的优化理论和严格的收敛保证为后盾。该研究将沿着以下协同方向发展：1)可扩展和随机GCCA算法；2)分布式、流化和非线性GCCA算法；3)验证，在遥感、脑成像、自然语言处理、传感器阵列处理等一系列及时而重要的应用。设计可扩展的、灵活的、流的和非线性的GCCA算法对于现代传感和分析问题是非常有动力的，这些问题涉及到快速增加的数据量和未知的潜在动态。将GCCA应用于大规模动态和复杂数据也提出了非常具有挑战性和令人兴奋的建模和优化问题。该奖项反映了美国国家科学基金会的法定使命，并通过使用基金会的知识价值和更广泛的影响审查标准进行评估，被认为值得支持。

Finding highly correlated latent factors in multimodal signals and data: Scalable algorithms and applications in sensing, imaging, and language processingAbstract: Multimodal signals and data arise naturally in many walks of science and engineering, and our digital society presents ever-increasing opportunities to collect and extract useful information from such data. For example, brain magnetic resonance imaging and electro-encephalography are two modes of sensing brain activity that can offer different "views" of the same set of patients (entities). Co-occurrence frequencies of a given set of words in different languages is another example. Crime, poverty, welfare, income, tax, school, unemployment, and other types of social data offer different views of a given set of municipalities. Integrating multiple views to extract meaningful common information is of great interest, and finds a vast amount of timely applications -- in brain imaging, machine translation, landscape change detection in remote sensing, and social science research, to name a few. However, existing multiview analytics tools -- notably (generalized) canonical correlation analysis [(G)CCA] -- are struggling to keep pace with the size of today's datasets, and the problem is only getting worse. Furthermore, the complex structure and dynamic nature of some of the underlying phenomena are not accounted for in classical GCCA. This project will provide much needed scalable and flexible computational tools for GCCA-based multimodal sensing and analytics, thereby benefiting a large variety of scientific and engineering applications. It will produce a framework allowing for plug-and-play incorporation of application-specific prior information, and distributed implementation. Beyond linear and batch GCCA, nonlinear GCCA and streaming GCCA will be considered. These are appealing and timely for many applications, but associated computational tools are sorely missing.In terms of theory and methods, many key aspects of GCCA (such as convergence properties, distributed implementation, and streaming variants) are still poorly understood. The research will provide a set of high-performance computational tools that are backed by advanced optimization theory and rigorous convergence guarantees. The research will evolve along the following synergistic thrusts: 1) scalable and stochastic GCCA algorithms; 2) distributed, streaming and nonlinear GCCA algorithms; and 3) validation, using a series of timely and important applications in remote sensing, brain imaging, natural language processing, and sensor array processing. Devising scalable, flexible, streaming, and nonlinear GCCA algorithms is very well-motivated for modern sensing and analytics problems which involve rapidly increasing amounts of data with unknown underlying dynamics. Using GCCA for large-scale dynamic and complex data also poses very challenging and exciting modeling and optimization problems.This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.